HeartSaVioR commented on pull request #29066: URL: https://github.com/apache/spark/pull/29066#issuecomment-690848690
This is definitely a major missing piece on DSv2 compared to DSv1, as DSv1 writer can deal with Dataframe directly, hence able to do arbitrary changes (including repartition/sort) before doing actual write, like I did for state data source - https://github.com/HeartSaVioR/spark-state-tools/blob/8a74bdb1bc7911a6f71785cf68b784b0a331a1d9/src/main/scala/net/heartsavior/spark/sql/state/StateStoreWriter.scala#L67-L74 Lots of data sources are blocked to migrate to DSv2. Shall we consider prioritizing this? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
