#general
@subbareddydagumati: @subbareddydagumati has joined the channel
@noahprince8: When using
@noahprince8: Also is there an easy way to just sink a Spark dataframe to pinot segments?
@noahprince8: Looks like I’d just need to manually use SegmentIndexCreationDriverImpl with a foreachpartition or something. Then manually do the copying that `SparkSegmentGenerationJobRunner` does. I might make a PR to just have a method that takes a dataframe, does a foreachpartition and maps it to GenericRow.
@noahprince8: Would be useful to have this feature since you can read from whatever custom spark readers you have, repartition as necessary, etc.
@jackie.jxt: @fx19880617 Thoughts?
@fx19880617: This is a useful feature
@fx19880617: I can help review the PR
@fx19880617: and let me know if you have any question on the segment gen task
@wa.moustafa: @wa.moustafa has joined the channel
#random
@subbareddydagumati: @subbareddydagumati has joined the channel
@wa.moustafa: @wa.moustafa has joined the channel
#new-office-space
@g.kishore: @g.kishore has joined the channel
@g.kishore: @g.kishore has left the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
