Github user koeninger commented on a diff in the pull request:
https://github.com/apache/spark/pull/3543#discussion_r22446683
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala
---
@@ -789,7 +790,7 @@ class JavaPairDStream[K, V](val dstream: DStream[(K,
V)])(
keyClass: Class[_],
valueClass: Class[_],
outputFormatClass: Class[_ <: NewOutputFormat[_, _]],
- conf: Configuration = new Configuration) {
--- End diff --
Based on what Marcelo Vanzin said on the dev list when I brought this issue
up, the only reason the problem was still around for me to run into is
because he changed some of the uses of new Configuration but not all of
them.
I agree it's used in a lot of different places, but I'm not sure how
piecemeal fixes to only some of the places is helpful to users. Were there
still specific concerns about particular classes?
On Sun, Jan 4, 2015 at 6:28 AM, Tathagata Das <[email protected]>
wrote:
> In
>
streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala
> <https://github.com/apache/spark/pull/3543#discussion-diff-22438855>:
>
> > @@ -789,7 +790,7 @@ class JavaPairDStream[K, V](val dstream:
DStream[(K, V)])(
> > keyClass: Class[_],
> > valueClass: Class[_],
> > outputFormatClass: Class[_ <: NewOutputFormat[_, _]],
> > - conf: Configuration = new Configuration) {
>
> The scope of this PR is pretty wide in terms of the number of classes it
> touches, causing issues as different places needs to be handled
> differently. If you considered moving this sort of changes (new
> Configuration to sparkContext.hadoopConfiguration) into a different PR
> that might be easier to get in.
>
> â
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/3543/files#r22438855>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
