Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/6150#discussion_r30365530
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
@@ -280,33 +280,42 @@ trait CatalystScan {
/**
* ::Experimental::
- * [[OutputWriter]] is used together with [[FSBasedRelation]] for
persisting rows to the
- * underlying file system. Subclasses of [[OutputWriter]] must provide a
zero-argument constructor.
- * An [[OutputWriter]] instance is created and initialized when a new
output file is opened on
- * executor side. This instance is used to persist rows to this single
output file.
+ * A factory that produces [[OutputWriter]]s. A new
[[OutputWriterFactory]] is created on driver
+ * side for each write job issued when writing to a [[HadoopFsRelation]],
and then gets serialized
+ * to executor side to create actual [[OutputWriter]]s on the fly.
*
* @since 1.4.0
*/
@Experimental
-abstract class OutputWriter {
+trait OutputWriterFactory extends Serializable {
--- End diff --
why trait over abstract class?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
