[jira] [Commented] (SPARK-25299) Use remote storage for persisting shuffle data

2018-12-26 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729357#comment-16729357 ] Saisai Shao commented on SPARK-25299: - [~jealous] Can we have a doc about this proposed solution for

[jira] [Updated] (SPARK-26449) Missing Dataframe.transform API in Python API

2018-12-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26449: - Summary: Missing Dataframe.transform API in Python API (was: Dataframe.transform) > Missing

[jira] [Resolved] (SPARK-26452) Suppressing exception in finally: Java heap space java.lang.OutOfMemoryError: Java heap space

2018-12-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26452. -- Resolution: Invalid Please don't just copy and paste the error message. Include information

[jira] [Commented] (SPARK-26449) Dataframe.transform

2018-12-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729301#comment-16729301 ] Hyukjin Kwon commented on SPARK-26449: -- Seems like Scala side has it but Python doesn't. Can you

[jira] [Updated] (SPARK-26449) Missing Dataframe.transform API in Python API

2018-12-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26449: - Issue Type: Improvement (was: New Feature) > Missing Dataframe.transform API in Python API >

[jira] [Updated] (SPARK-26449) Missing Dataframe.transform API in Python API

2018-12-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26449: - Labels: (was: patch) > Missing Dataframe.transform API in Python API >

[jira] [Updated] (SPARK-26449) Missing Dataframe.transform API in Python API

2018-12-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26449: - Component/s: PySpark > Missing Dataframe.transform API in Python API >

[jira] [Commented] (SPARK-26438) Driver waits to spark.sql.broadcastTimeout before throwing OutOfMemoryError - is this by design?

2018-12-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729300#comment-16729300 ] Hyukjin Kwon commented on SPARK-26438: -- Let's ask a question to Spark mailing list before filing an

[jira] [Resolved] (SPARK-26438) Driver waits to spark.sql.broadcastTimeout before throwing OutOfMemoryError - is this by design?

2018-12-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26438. -- Resolution: Invalid > Driver waits to spark.sql.broadcastTimeout before throwing

[jira] [Commented] (SPARK-26268) Decouple shuffle data from Spark deployment

2018-12-26 Thread Peiyu Zhuang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729297#comment-16729297 ] Peiyu Zhuang commented on SPARK-26268: -- Check

[jira] [Comment Edited] (SPARK-25299) Use remote storage for persisting shuffle data

2018-12-26 Thread Peiyu Zhuang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729292#comment-16729292 ] Peiyu Zhuang edited comment on SPARK-25299 at 12/27/18 3:31 AM: We are

[jira] [Commented] (SPARK-25299) Use remote storage for persisting shuffle data

2018-12-26 Thread Carson Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729294#comment-16729294 ] Carson Wang commented on SPARK-25299: - I am on a vacation and will be back on January 2, 2019.

[jira] [Commented] (SPARK-25299) Use remote storage for persisting shuffle data

2018-12-26 Thread Peiyu Zhuang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729292#comment-16729292 ] Peiyu Zhuang commented on SPARK-25299: -- We are currently working on a solution that is similar to

[jira] [Resolved] (SPARK-26424) Use java.time API in timestamp/date functions

2018-12-26 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-26424. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23358

[jira] [Assigned] (SPARK-26424) Use java.time API in timestamp/date functions

2018-12-26 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-26424: --- Assignee: Maxim Gekk > Use java.time API in timestamp/date functions >

[jira] [Created] (SPARK-26452) Suppressing exception in finally: Java heap space java.lang.OutOfMemoryError: Java heap space

2018-12-26 Thread tommy duan (JIRA)
tommy duan created SPARK-26452: -- Summary: Suppressing exception in finally: Java heap space java.lang.OutOfMemoryError: Java heap space Key: SPARK-26452 URL: https://issues.apache.org/jira/browse/SPARK-26452

[jira] [Updated] (SPARK-26439) Introduce WorkerOffer reservation mechanism for Barrier TaskSet

2018-12-26 Thread wuyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-26439: - Description: Currently, Barrier TaskSet has a hard requirement that tasks can only be launched in a single

[jira] [Commented] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-12-26 Thread Jacky Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729261#comment-16729261 ] Jacky Li commented on SPARK-24630: -- Actually I encountered this scenario earlier, so we have

[jira] [Updated] (SPARK-26439) Introduce WorkerOffer reservation mechanism for Barrier TaskSet

2018-12-26 Thread wuyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-26439: - Summary: Introduce WorkerOffer reservation mechanism for Barrier TaskSet (was: Introduce WorkOffer reservation

[jira] [Created] (SPARK-26451) Change lead/lag argument name from count to offset

2018-12-26 Thread Deepyaman Datta (JIRA)
Deepyaman Datta created SPARK-26451: --- Summary: Change lead/lag argument name from count to offset Key: SPARK-26451 URL: https://issues.apache.org/jira/browse/SPARK-26451 Project: Spark

[jira] [Updated] (SPARK-26378) Queries of wide CSV/JSON data slowed after SPARK-26151

2018-12-26 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-26378: -- Description: A recent change significantly slowed the queries of wide CSV tables. For

[jira] [Updated] (SPARK-26378) Queries of wide CSV/JSON data slowed after SPARK-26151

2018-12-26 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-26378: -- Summary: Queries of wide CSV/JSON data slowed after SPARK-26151 (was: Queries of wide CSV

[jira] [Assigned] (SPARK-26451) Change lead/lag argument name from count to offset

2018-12-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26451: Assignee: (was: Apache Spark) > Change lead/lag argument name from count to offset >

[jira] [Assigned] (SPARK-26451) Change lead/lag argument name from count to offset

2018-12-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26451: Assignee: Apache Spark > Change lead/lag argument name from count to offset >

[jira] [Created] (SPARK-26450) Map of schema is built too frequently in some wide queries

2018-12-26 Thread Bruce Robbins (JIRA)
Bruce Robbins created SPARK-26450: - Summary: Map of schema is built too frequently in some wide queries Key: SPARK-26450 URL: https://issues.apache.org/jira/browse/SPARK-26450 Project: Spark

[jira] [Updated] (SPARK-26449) Dataframe.transform

2018-12-26 Thread Hanan Shteingart (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanan Shteingart updated SPARK-26449: - Shepherd: Maciej Szymkiewicz (was: Lazy Developer) > Dataframe.transform >

[jira] [Updated] (SPARK-26449) Dataframe.transform

2018-12-26 Thread Hanan Shteingart (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanan Shteingart updated SPARK-26449: - Shepherd: Lazy Developer (was: yc) > Dataframe.transform > --- > >

[jira] [Created] (SPARK-26449) Dataframe.transform

2018-12-26 Thread Hanan Shteingart (JIRA)
Hanan Shteingart created SPARK-26449: Summary: Dataframe.transform Key: SPARK-26449 URL: https://issues.apache.org/jira/browse/SPARK-26449 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-23959) UnresolvedException with DataSet created from Seq.empty since Spark 2.3.0

2018-12-26 Thread Sam hendley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729148#comment-16729148 ] Sam hendley commented on SPARK-23959: - I am working on upgrading a medium sized project from spark

[jira] [Assigned] (SPARK-26448) retain the difference between 0.0 and -0.0

2018-12-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26448: Assignee: Wenchen Fan (was: Apache Spark) > retain the difference between 0.0 and -0.0

[jira] [Assigned] (SPARK-26448) retain the difference between 0.0 and -0.0

2018-12-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26448: Assignee: Apache Spark (was: Wenchen Fan) > retain the difference between 0.0 and -0.0

[jira] [Created] (SPARK-26448) retain the difference between 0.0 and -0.0

2018-12-26 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-26448: --- Summary: retain the difference between 0.0 and -0.0 Key: SPARK-26448 URL: https://issues.apache.org/jira/browse/SPARK-26448 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-26447) Allow OrcColumnarBatchReader to return less partition columns

2018-12-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26447: Assignee: (was: Apache Spark) > Allow OrcColumnarBatchReader to return less

[jira] [Assigned] (SPARK-26447) Allow OrcColumnarBatchReader to return less partition columns

2018-12-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26447: Assignee: Apache Spark > Allow OrcColumnarBatchReader to return less partition columns >

[jira] [Created] (SPARK-26447) Allow OrcColumnarBatchReader to return less partition columns

2018-12-26 Thread Gengliang Wang (JIRA)
Gengliang Wang created SPARK-26447: -- Summary: Allow OrcColumnarBatchReader to return less partition columns Key: SPARK-26447 URL: https://issues.apache.org/jira/browse/SPARK-26447 Project: Spark

[jira] [Assigned] (SPARK-26446) improve doc on ExecutorAllocationManager

2018-12-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26446: Assignee: Apache Spark > improve doc on ExecutorAllocationManager >

[jira] [Assigned] (SPARK-26446) improve doc on ExecutorAllocationManager

2018-12-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26446: Assignee: (was: Apache Spark) > improve doc on ExecutorAllocationManager >

[jira] [Updated] (SPARK-26446) improve doc on ExecutorAllocationManager

2018-12-26 Thread Qingxin Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingxin Wu updated SPARK-26446: --- Component/s: (was: Scheduler) Spark Core > improve doc on

[jira] [Created] (SPARK-26446) improve doc on ExecutorAllocationManager

2018-12-26 Thread Qingxin Wu (JIRA)
Qingxin Wu created SPARK-26446: -- Summary: improve doc on ExecutorAllocationManager Key: SPARK-26446 URL: https://issues.apache.org/jira/browse/SPARK-26446 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-26445) Use ConfigEntry for hardcoded configs for driver/executor categories.

2018-12-26 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-26445: - Summary: Use ConfigEntry for hardcoded configs for driver/executor categories. Key: SPARK-26445 URL: https://issues.apache.org/jira/browse/SPARK-26445 Project:

[jira] [Commented] (SPARK-26445) Use ConfigEntry for hardcoded configs for driver/executor categories.

2018-12-26 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16728974#comment-16728974 ] Takuya Ueshin commented on SPARK-26445: --- I'm working on this. > Use ConfigEntry for hardcoded

[jira] [Assigned] (SPARK-26444) Stage color doesn't change with it's status

2018-12-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26444: Assignee: Apache Spark > Stage color doesn't change with it's status >

[jira] [Assigned] (SPARK-26444) Stage color doesn't change with it's status

2018-12-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26444: Assignee: (was: Apache Spark) > Stage color doesn't change with it's status >

[jira] [Updated] (SPARK-26444) Stage color doesn't change with it's status

2018-12-26 Thread Chenxiao Mao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chenxiao Mao updated SPARK-26444: - Attachment: failed.png complete.png active.png > Stage color

[jira] [Updated] (SPARK-26444) Stage color doesn't change with it's status

2018-12-26 Thread Chenxiao Mao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chenxiao Mao updated SPARK-26444: - Description: On job page, in event timeline section, stage color doesn't change according to

[jira] [Created] (SPARK-26444) Stage color doesn't change with it's status

2018-12-26 Thread Chenxiao Mao (JIRA)
Chenxiao Mao created SPARK-26444: Summary: Stage color doesn't change with it's status Key: SPARK-26444 URL: https://issues.apache.org/jira/browse/SPARK-26444 Project: Spark Issue Type: Bug