[jira] [Commented] (SPARK-2868) Support named accumulators in Python
[ https://issues.apache.org/jira/browse/SPARK-2868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17512540#comment-17512540 ] Rafal Wojdyla commented on SPARK-2868: -- Is there a better issue to track the work on named accumulators in pyspark? Is it still the case that named accumulators do not work in pyspark and it's not possible to see pyspark accumulators in the web UI? Would appreciate you feedback [~pwendell] [~holden] [~heathkh] please? > Support named accumulators in Python > > > Key: SPARK-2868 > URL: https://issues.apache.org/jira/browse/SPARK-2868 > Project: Spark > Issue Type: New Feature > Components: PySpark >Reporter: Patrick Wendell >Priority: Major > Labels: bulk-closed > > SPARK-2380 added this for Java/Scala. To allow this in Python we'll need to > make some additional changes. One potential path is to have a 1:1 > correspondence with Scala accumulators (instead of a one-to-many). A > challenge is exposing the stringified values of the accumulators to the Scala > code. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2868) Support named accumulators in Python
[ https://issues.apache.org/jira/browse/SPARK-2868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16041557#comment-16041557 ] Kyle Heath commented on SPARK-2868: --- @[~holdenk]: I would love to better understand the scope of the work if you have time to sketch it out for me. > Support named accumulators in Python > > > Key: SPARK-2868 > URL: https://issues.apache.org/jira/browse/SPARK-2868 > Project: Spark > Issue Type: New Feature > Components: PySpark >Reporter: Patrick Wendell > > SPARK-2380 added this for Java/Scala. To allow this in Python we'll need to > make some additional changes. One potential path is to have a 1:1 > correspondence with Scala accumulators (instead of a one-to-many). A > challenge is exposing the stringified values of the accumulators to the Scala > code. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2868) Support named accumulators in Python
[ https://issues.apache.org/jira/browse/SPARK-2868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849723#comment-15849723 ] holdenk commented on SPARK-2868: This might be a difficult issue to start of with [~heathkh] - the accumulator API has been changed a lot on the backend and the Python API is still exposing the old API. If your interested I might start by picking a smaller PySpark related issue and then coming back to this. I'd be happy to chat with you though about what the work involved might look like for this issue. > Support named accumulators in Python > > > Key: SPARK-2868 > URL: https://issues.apache.org/jira/browse/SPARK-2868 > Project: Spark > Issue Type: New Feature > Components: PySpark >Reporter: Patrick Wendell > > SPARK-2380 added this for Java/Scala. To allow this in Python we'll need to > make some additional changes. One potential path is to have a 1:1 > correspondence with Scala accumulators (instead of a one-to-many). A > challenge is exposing the stringified values of the accumulators to the Scala > code. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2868) Support named accumulators in Python
[ https://issues.apache.org/jira/browse/SPARK-2868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15827519#comment-15827519 ] Kyle Heath commented on SPARK-2868: --- Short version: Is there anything I can do to help bring this feature to pyspark? Long version: I've been implementing large jobs in pyspark for about 6 months. The ability to monitor named accumulators in the web-ui seems really important. Running complex jobs at scale has been a bit like flying blind. I'm new to this community, but want to help if I can. > Support named accumulators in Python > > > Key: SPARK-2868 > URL: https://issues.apache.org/jira/browse/SPARK-2868 > Project: Spark > Issue Type: New Feature > Components: PySpark >Reporter: Patrick Wendell > > SPARK-2380 added this for Java/Scala. To allow this in Python we'll need to > make some additional changes. One potential path is to have a 1:1 > correspondence with Scala accumulators (instead of a one-to-many). A > challenge is exposing the stringified values of the accumulators to the Scala > code. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2868) Support named accumulators in Python
[ https://issues.apache.org/jira/browse/SPARK-2868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15625621#comment-15625621 ] holdenk commented on SPARK-2868: or maybe [~rxin] or [~squito] who have been doing some other accumulator work? > Support named accumulators in Python > > > Key: SPARK-2868 > URL: https://issues.apache.org/jira/browse/SPARK-2868 > Project: Spark > Issue Type: New Feature > Components: PySpark >Reporter: Patrick Wendell > > SPARK-2380 added this for Java/Scala. To allow this in Python we'll need to > make some additional changes. One potential path is to have a 1:1 > correspondence with Scala accumulators (instead of a one-to-many). A > challenge is exposing the stringified values of the accumulators to the Scala > code. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2868) Support named accumulators in Python
[ https://issues.apache.org/jira/browse/SPARK-2868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15579866#comment-15579866 ] holdenk commented on SPARK-2868: ping [~davies] - would you be available to review if I got this switched around? > Support named accumulators in Python > > > Key: SPARK-2868 > URL: https://issues.apache.org/jira/browse/SPARK-2868 > Project: Spark > Issue Type: New Feature > Components: PySpark >Reporter: Patrick Wendell > > SPARK-2380 added this for Java/Scala. To allow this in Python we'll need to > make some additional changes. One potential path is to have a 1:1 > correspondence with Scala accumulators (instead of a one-to-many). A > challenge is exposing the stringified values of the accumulators to the Scala > code. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2868) Support named accumulators in Python
[ https://issues.apache.org/jira/browse/SPARK-2868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556582#comment-15556582 ] holdenk commented on SPARK-2868: Is this something we are still interested in pursuing (cc [~rxin] who did the Scala accumulator API update). I'd be happy to take on this issue, once we've decide what to do around data property accumulators, since I've already been working with accumulators a bunch. > Support named accumulators in Python > > > Key: SPARK-2868 > URL: https://issues.apache.org/jira/browse/SPARK-2868 > Project: Spark > Issue Type: New Feature > Components: PySpark >Reporter: Patrick Wendell > > SPARK-2380 added this for Java/Scala. To allow this in Python we'll need to > make some additional changes. One potential path is to have a 1:1 > correspondence with Scala accumulators (instead of a one-to-many). A > challenge is exposing the stringified values of the accumulators to the Scala > code. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
