[
https://issues.apache.org/jira/browse/SPARK-7591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheng Lian resolved SPARK-7591.
-------------------------------
Resolution: Fixed
Fix Version/s: 1.4.0
Issue resolved by pull request 6150
[https://github.com/apache/spark/pull/6150]
> FSBasedRelation interface tweaks
> --------------------------------
>
> Key: SPARK-7591
> URL: https://issues.apache.org/jira/browse/SPARK-7591
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Reporter: Reynold Xin
> Assignee: Cheng Lian
> Priority: Blocker
> Fix For: 1.4.0
>
>
> # Renaming {{FSBasedRelation}} to {{HadoopFsRelation}}
> Since itss all coupled with Hadoop {{FileSystem}} and job API.
> # {{HadoopFsRelation}} should have a no-arg constructor
> {{paths}} and {{partitionColumns}} should just be methods to be overridden,
> rather than constructor arguments. This makes data source developers life
> easier by having a no-arg constructor and being serialization friendly.
> # Renaming {{HadoopFsRelation.prepareForWrite}} to
> {{HadoopFsRelation.prepareJobForWrite}}
> The new name explicitly suggests developers should only touch the {{Job}}
> instance for preparation work (which is also documented in Scaladoc).
> # Allowing serialization while creating {{OutputWriter}}s
> To be more precise, {{OutputWriter}}s are never created on driver side and
> serialized to executor side. But the factory that creates {{OutputWriter}}s
> should be created on driver side and serialized.
> The reason behind this is that, passing all needed materials to
> {{OutputWriter}} instances via Hadoop Configuration is doable but sometimes
> neither intuitive nor convenient. Resorting to serialization makes data
> source developers' life easier. Actually this happens when I was migrating
> the Parquet data source, and wanted to pass the final output path (instead of
> temporary work path) to the output writer (see
> [here|https://github.com/liancheng/spark/commit/ec9950c591e5b981ce20fab96562db28488e0035#diff-53521d336f7259e859fea4d3ca4dc888R74]).
> There I have to put a property into the Configuration object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
