[
https://issues.apache.org/jira/browse/SPARK-61646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61646:
-----------------------------------
Labels: pull-request-available (was: )
> Support write distribution and ordering in the Python Data Source API
> ---------------------------------------------------------------------
>
> Key: SPARK-61646
> URL: https://issues.apache.org/jira/browse/SPARK-61646
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 4.4.0
> Reporter: Ganesha S
> Priority: Major
> Labels: pull-request-available
>
> Python data source writers have no way to tell Spark how their input rows
> should be grouped or sorted. Scala/Java (DSv2) sinks can already do this
> through RequiresDistributionAndOrdering, and Spark adds the needed shuffle
> and sort before the write.
> Without this, a Python sink that writes partitioned or sorted output gets
> rows in whatever layout the query produced. The result is many small files or
> requests, or the sink has to ask users to call repartition() /
> sortWithinPartitions() themselves.
> Proposal: add optional methods to DataSourceWriter so a writer can declare
> its requirements:
>
> {code:java}
> class MyWriter(DataSourceWriter):
> def requiredDistribution(self):
> return ClusteredDistribution(["region"])
> def requiredOrdering(self):
> return [SortOrder("event_time")]{code}
>
> Spark asks the writer once while planning the query and adds the repartition
> and sort automatically, using the same planner path as DSv2 sinks.
> - Optional methods: requiredDistribution, requiredOrdering,
> requiredNumPartitions, distributionStrictlyRequired,
> advisoryPartitionSizeInBytes.
> - Every method defaults to "no requirement", so existing writers are
> unaffected.
> - Gated by an internal config,
> spark.sql.python.writeDistributionAndOrdering.enabled, off by default.
> Related: SPARK-56635 (read side: letting a Python source report its existing
> partitioning).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]