Github user koeninger commented on a diff in the pull request:
https://github.com/apache/spark/pull/3543#discussion_r21610025
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -262,7 +263,7 @@ class SQLContext(@transient val sparkContext:
SparkContext)
def createParquetFile[A <: Product : TypeTag](
path: String,
allowExisting: Boolean = true,
- conf: Configuration = new Configuration()): SchemaRDD = {
--- End diff --
I seem to recall there being potential thread safety issues related to
hadoop configuration objects, resulting in the need to create / clone them.
Quick search turned up e.g.
https://issues.apache.org/jira/browse/SPARK-2546
I'm not sure how relevant that is to all of these existing situations where
new Configuration() is being called.
On Tue, Dec 9, 2014 at 5:07 PM, Tathagata Das <[email protected]>
wrote:
> In sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala
> <https://github.com/apache/spark/pull/3543#discussion-diff-21571141>:
>
> > @@ -262,7 +263,7 @@ class SQLContext(@transient val sparkContext:
SparkContext)
> > def createParquetFile[A <: Product : TypeTag](
> > path: String,
> > allowExisting: Boolean = true,
> > - conf: Configuration = new Configuration()): SchemaRDD = {
>
> I think this should be using the "hadoopConfiguration" object in the
> SparkContext. That has all the hadoop related configuration already setup
> and should be what is automatically used. @marmbrus
> <https://github.com/marmbrus> should have a better idea.
>
> â
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/3543/files#r21571141>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
