[
https://issues.apache.org/jira/browse/SPARK-8135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14575400#comment-14575400
]
Josh Rosen commented on SPARK-8135:
-----------------------------------
Just curious: do we necessarily need to use ObjectWritable instances here? Can
we we get away with calling writeFields() directly on the Writable instances
and using reflection to instantiate the class from its name using its default
constructor?
> In SerializableWritable, don't load defaults when instantiating Configuration
> -----------------------------------------------------------------------------
>
> Key: SPARK-8135
> URL: https://issues.apache.org/jira/browse/SPARK-8135
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Reporter: Sandy Ryza
> Assignee: Sandy Ryza
>
> Calling "new Configuration()" is an expensive operation because it loads any
> Hadoop configuration XMLs from disk.
> In SerializableWritable, we call new Configuration needlessly when
> instantiating an ObjectWritable. The ObjectWritable only needs the
> Configuration for its class cache, not for any Hadoop properties that might
> be in XML files, so it should be ok to call new Configuration with
> loadDefaults = false.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
