[ https://issues.apache.org/jira/browse/SPARK-7591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheng Lian resolved SPARK-7591. ------------------------------- Resolution: Fixed Fix Version/s: 1.4.0 Issue resolved by pull request 6150 [https://github.com/apache/spark/pull/6150] > FSBasedRelation interface tweaks > -------------------------------- > > Key: SPARK-7591 > URL: https://issues.apache.org/jira/browse/SPARK-7591 > Project: Spark > Issue Type: Sub-task > Components: SQL > Reporter: Reynold Xin > Assignee: Cheng Lian > Priority: Blocker > Fix For: 1.4.0 > > > # Renaming {{FSBasedRelation}} to {{HadoopFsRelation}} > Since itss all coupled with Hadoop {{FileSystem}} and job API. > # {{HadoopFsRelation}} should have a no-arg constructor > {{paths}} and {{partitionColumns}} should just be methods to be overridden, > rather than constructor arguments. This makes data source developers life > easier by having a no-arg constructor and being serialization friendly. > # Renaming {{HadoopFsRelation.prepareForWrite}} to > {{HadoopFsRelation.prepareJobForWrite}} > The new name explicitly suggests developers should only touch the {{Job}} > instance for preparation work (which is also documented in Scaladoc). > # Allowing serialization while creating {{OutputWriter}}s > To be more precise, {{OutputWriter}}s are never created on driver side and > serialized to executor side. But the factory that creates {{OutputWriter}}s > should be created on driver side and serialized. > The reason behind this is that, passing all needed materials to > {{OutputWriter}} instances via Hadoop Configuration is doable but sometimes > neither intuitive nor convenient. Resorting to serialization makes data > source developers' life easier. Actually this happens when I was migrating > the Parquet data source, and wanted to pass the final output path (instead of > temporary work path) to the output writer (see > [here|https://github.com/liancheng/spark/commit/ec9950c591e5b981ce20fab96562db28488e0035#diff-53521d336f7259e859fea4d3ca4dc888R74]). > There I have to put a property into the Configuration object. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org