ulysses-you commented on code in PR #39277:
URL: https://github.com/apache/spark/pull/39277#discussion_r1062020277
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/WriteFiles.scala:
##########
@@ -53,13 +59,17 @@ case class WriteFiles(child: LogicalPlan) extends UnaryNode
{
/**
* Responsible for writing files.
*/
-case class WriteFilesExec(child: SparkPlan) extends UnaryExecNode {
+case class WriteFilesExec(
+ child: SparkPlan,
+ fileFormat: FileFormat,
+ partitionColumns: Seq[Attribute],
+ bucketSpec: Option[BucketSpec],
+ options: Map[String, String],
+ staticPartitions: TablePartitionSpec) extends UnaryExecNode {
override def output: Seq[Attribute] = Seq.empty
- override protected def doExecuteWrite(writeSpec: WriteSpec):
RDD[WriterCommitMessage] = {
- assert(writeSpec.isInstanceOf[WriteFilesSpec])
- val writeFilesSpec: WriteFilesSpec = writeSpec.asInstanceOf[WriteFilesSpec]
-
+ override protected def doExecuteWrite(
+ writeFilesSpec: WriteFilesSpec): RDD[WriterCommitMessage] = {
Review Comment:
Seems it's a bit hard. look at the current information:
```scala
case class WriteFilesSpec(
description: WriteJobDescription,
committer: FileCommitProtocol,
concurrentOutputWriterSpecFunc: SparkPlan =>
Option[ConcurrentOutputWriterSpec])
```
- `ConcurrentOutputWriterSpec` and `FileCommitProtocol` contain the output
spec so we can not replace them
- `WriteJobDescription` contains many information which includes what we
pull out, but if we want to reduce something inside `WriteJobDescription`, we
need to create a new class to hold others. I'm not sure it's worth to do that.
```scala
class WriteJobDescription(
val uuid: String,
val serializableHadoopConf: SerializableConfiguration,
val outputWriterFactory: OutputWriterFactory,
val allColumns: Seq[Attribute],
val dataColumns: Seq[Attribute],
val partitionColumns: Seq[Attribute],
val bucketSpec: Option[WriterBucketSpec],
val path: String,
val customPartitionLocations: Map[TablePartitionSpec, String],
val maxRecordsPerFile: Long,
val timeZoneId: String,
val statsTrackers: Seq[WriteJobStatsTracker])
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
