[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16578856#comment-16578856
]
Imran Rashid commented on SPARK-650:
Folks may be interested in SPARK-24918. perhaps
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16517398#comment-16517398
]
Avi minsky commented on SPARK-650:
--
We encountered an issue with the combination of lazy
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511033#comment-16511033
]
Sina Madani commented on SPARK-650:
---
I too have this problem. It seems that Apache Flink
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16239991#comment-16239991
]
quang nguyen commented on SPARK-650:
Hi,
We had an application run on spark cluster to
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158202#comment-16158202
]
yiming.xu commented on SPARK-650:
-
I need a hook too. Some case, We need init something lik
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117114#comment-16117114
]
Sean Owen commented on SPARK-650:
-
I can also imagine cases involving legacy code that make
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117090#comment-16117090
]
Louis Bergelson commented on SPARK-650:
---
[~srowen] Thanks for the reply and the exam
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110506#comment-16110506
]
Sean Owen commented on SPARK-650:
-
Are you looking for an example of how it works? somethin
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110006#comment-16110006
]
Michael Schmeißer commented on SPARK-650:
-
Please see my comment from 05/Dec/16 12:
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109311#comment-16109311
]
Sean Owen commented on SPARK-650:
-
I still don't see an argument against my primary suggest
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109179#comment-16109179
]
Louis Bergelson commented on SPARK-650:
---
I can't understand how people are dismissing
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16054162#comment-16054162
]
Michael Schmeißer commented on SPARK-650:
-
[~riteshtijoriwala] - Sorry, but I am no
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16053745#comment-16053745
]
Ritesh Tijoriwala commented on SPARK-650:
-
[~Skamandros] - Any similar tricks for s
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15976902#comment-15976902
]
Michael Schmeißer commented on SPARK-650:
-
In a nutshell, we have our own class "My
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15969422#comment-15969422
]
Ritesh Tijoriwala commented on SPARK-650:
-
[~Skamandros] - I would also like to kno
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967554#comment-15967554
]
Mathieu Boespflug commented on SPARK-650:
-
[~Skamandros] how did you manage to hook
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15963304#comment-15963304
]
Ryan Williams commented on SPARK-650:
-
Both suggested workarounds here are lacking or b
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731628#comment-15731628
]
Michael Schmeißer commented on SPARK-650:
-
No, it's not just about propagating info
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15725790#comment-15725790
]
Herman van Hovell commented on SPARK-650:
-
A creatively applied broadcast variable
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15725784#comment-15725784
]
Herman van Hovell commented on SPARK-650:
-
If you only try to propagate information
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15725542#comment-15725542
]
Michael Schmeißer commented on SPARK-650:
-
Sure it can be included in the closure a
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15723170#comment-15723170
]
Sean Owen commented on SPARK-650:
-
Why? info X can be included in the closure, and the exec
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15722178#comment-15722178
]
Michael Schmeißer commented on SPARK-650:
-
Thanks [~robert.neumann]! I am ready to
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15722170#comment-15722170
]
Michael Schmeißer commented on SPARK-650:
-
A singleton is not really feasible if ad
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15714957#comment-15714957
]
Robert Neumann commented on SPARK-650:
--
OK. Will do.
> Add a "setup hook" API for run
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15714819#comment-15714819
]
Herman van Hovell commented on SPARK-650:
-
[~lars_francke][~Skamandros][~rneumann]
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15714762#comment-15714762
]
Robert Neumann commented on SPARK-650:
--
Sean, I agree this is the essential question i
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15714760#comment-15714760
]
Robert Neumann commented on SPARK-650:
--
Sean, I agree this is the essential question i
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15714725#comment-15714725
]
Sean Owen commented on SPARK-650:
-
Why would a singleton not work? This is really the essen
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15714611#comment-15714611
]
Robert Neumann commented on SPARK-650:
--
I am supporting Olivier Armand. We need a way
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15581982#comment-15581982
]
Sean Owen commented on SPARK-650:
-
Yep, if you must pass some configuration, it generally c
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15581971#comment-15581971
]
Michael Schmeißer commented on SPARK-650:
-
I agree that static initialization would
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15581774#comment-15581774
]
Sean Owen commented on SPARK-650:
-
BTW I am not suggesting an "empty RDD" for your case. Th
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15580502#comment-15580502
]
Michael Schmeißer commented on SPARK-650:
-
What if I have a Hadoop InputFormat? The
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15580135#comment-15580135
]
Sean Owen commented on SPARK-650:
-
But, why do you need to do it before you have an RDD? Yo
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15580069#comment-15580069
]
Michael Schmeißer commented on SPARK-650:
-
But I'll need to have an RDD to do this,
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15580062#comment-15580062
]
Sean Owen commented on SPARK-650:
-
This is still easy to do with mapPartitions, which can c
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15580055#comment-15580055
]
Michael Schmeißer commented on SPARK-650:
-
Ok, let me explain the specific problems
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15579928#comment-15579928
]
Sean Owen commented on SPARK-650:
-
Yeah that's a decent use case, because latency is an iss
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15579737#comment-15579737
]
Olivier Armand commented on SPARK-650:
--
Data doesn't arrives necessarily immediately,
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15579720#comment-15579720
]
Sean Owen commented on SPARK-650:
-
It would work in this case to immediately schedule initi
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15579710#comment-15579710
]
Olivier Armand commented on SPARK-650:
--
> "just run a dummy mapPartitions at the outse
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15579630#comment-15579630
]
Sean Owen commented on SPARK-650:
-
Reopening doesn't do anything by itself, or cause anyone
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15578914#comment-15578914
]
Lars Francke commented on SPARK-650:
I can only come up with three reasons at the momen
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15578875#comment-15578875
]
Lars Francke commented on SPARK-650:
I also have to disagree with this being a duplicat
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15578535#comment-15578535
]
Sean Owen commented on SPARK-650:
-
If you need init to happen ASAP when the driver starts,
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15578487#comment-15578487
]
Olivier Armand commented on SPARK-650:
--
Sean, a singleton is not the best option in ou
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15578378#comment-15578378
]
Sean Owen commented on SPARK-650:
-
Sorry, I mean the _status_ doesn't matter. Most issues t
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15578361#comment-15578361
]
Michael Schmeißer commented on SPARK-650:
-
Then somebody should please explain to m
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15578338#comment-15578338
]
Sean Owen commented on SPARK-650:
-
In practice, these should probably all be WontFix as it
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15578289#comment-15578289
]
Michael Schmeißer commented on SPARK-650:
-
I disagree that those issues are duplica
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573360#comment-15573360
]
holdenk commented on SPARK-650:
---
Would people feel ok if we marked this as a duplicate of 636
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15561958#comment-15561958
]
Michael Schmeißer commented on SPARK-650:
-
To mee, the two seem related, but not ex
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15556324#comment-15556324
]
holdenk commented on SPARK-650:
---
I think this is a duplicate of SPARK-636 yes?
> Add a "setu
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15554861#comment-15554861
]
Luis Ramos commented on SPARK-650:
--
I have similar requirements to Michael's – this would
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14916264#comment-14916264
]
Michael Schmeißer commented on SPARK-650:
-
I would need this feature as well to per
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14645919#comment-14645919
]
Lars Francke commented on SPARK-650:
Not [~matei] but I think this would be a good idea
[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14206182#comment-14206182
]
Andrew Ash commented on SPARK-650:
--
As mentioned in SPARK-572 static classes' initializati
58 matches
Mail list logo