[jira] [Commented] (SPARK-29038) SPIP: Support Spark Materialized View

2019-09-10 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927256#comment-16927256 ] Lantao Jin commented on SPARK-29038: [~angerszhuuu]Of course, will contact you offline > SPIP:

[jira] [Updated] (SPARK-29033) Always use CreateNamedStructUnsafe, the UnsafeRow-based version of the CreateNamedStruct codepath

2019-09-10 Thread Josh Rosen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-29033: --- Summary: Always use CreateNamedStructUnsafe, the UnsafeRow-based version of the CreateNamedStruct

[jira] [Commented] (SPARK-29038) SPIP: Support Spark Materialized View

2019-09-10 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927274#comment-16927274 ] Xiao Li commented on SPARK-29038: - https://www.bwdb2ug.org/Presentations/BWDUG_%20MQT.pps is a

[jira] [Commented] (SPARK-29038) SPIP: Support Spark Materialized View

2019-09-10 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927268#comment-16927268 ] Xiao Li commented on SPARK-29038: - We need to follow ANSI SQL if we plan to support the materialized

[jira] [Commented] (SPARK-29038) SPIP: Support Spark Materialized View

2019-09-10 Thread Dilip Biswal (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927270#comment-16927270 ] Dilip Biswal commented on SPARK-29038: -- [~jerryshao] [~smilegator] Thanks.  > SPIP: Support Spark

[jira] [Commented] (SPARK-29038) SPIP: Support Spark Materialized View

2019-09-10 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927229#comment-16927229 ] Lantao Jin commented on SPARK-29038: [~angerszhuuu] By default, we use Parquet to storage the data

[jira] [Commented] (SPARK-29038) SPIP: Support Spark Materialized View

2019-09-10 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927243#comment-16927243 ] angerszhu commented on SPARK-29038: --- I ma interested in the match about : you create a MV table 

[jira] [Comment Edited] (SPARK-29038) SPIP: Support Spark Materialized View

2019-09-10 Thread Dilip Biswal (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927258#comment-16927258 ] Dilip Biswal edited comment on SPARK-29038 at 9/11/19 5:13 AM: ---

[jira] [Commented] (SPARK-29038) SPIP: Support Spark Materialized View

2019-09-10 Thread Saisai Shao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927265#comment-16927265 ] Saisai Shao commented on SPARK-29038: - [~cltlfcjin] I think we need a SPIP review and vote on the

[jira] [Comment Edited] (SPARK-29038) SPIP: Support Spark Materialized View

2019-09-10 Thread Dilip Biswal (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927258#comment-16927258 ] Dilip Biswal edited comment on SPARK-29038 at 9/11/19 5:09 AM: ---

[jira] [Comment Edited] (SPARK-29038) SPIP: Support Spark Materialized View

2019-09-10 Thread Dilip Biswal (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927258#comment-16927258 ] Dilip Biswal edited comment on SPARK-29038 at 9/11/19 5:09 AM: ---

[jira] [Commented] (SPARK-29038) SPIP: Support Spark Materialized View

2019-09-10 Thread Dilip Biswal (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927258#comment-16927258 ] Dilip Biswal commented on SPARK-29038: -- [~cltlfcjin] Actually i had similar question as

[jira] [Commented] (SPARK-29038) SPIP: Support Spark Materialized View

2019-09-10 Thread Saisai Shao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927263#comment-16927263 ] Saisai Shao commented on SPARK-29038: - IIUC, I think the key difference between MV and Spark's

[jira] [Issue Comment Deleted] (SPARK-29022) SparkSQLCLI can not use 'ADD JAR' 's jar as Serder class

2019-09-10 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-29022: -- Comment: was deleted (was: PR [https://github.com/apache/spark/pull/25729]) > SparkSQLCLI can not

[jira] [Created] (SPARK-29036) SparkThriftServer may can't cancel job after call a cancel before start.

2019-09-10 Thread angerszhu (Jira)
angerszhu created SPARK-29036: - Summary: SparkThriftServer may can't cancel job after call a cancel before start. Key: SPARK-29036 URL: https://issues.apache.org/jira/browse/SPARK-29036 Project: Spark

[jira] [Comment Edited] (SPARK-29009) Returning pojo from udf not working

2019-09-10 Thread Tomasz Belina (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926554#comment-16926554 ] Tomasz Belina edited comment on SPARK-29009 at 9/10/19 12:03 PM: - POJO

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-10 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926577#comment-16926577 ] koert kuipers commented on SPARK-29027: --- i am running test on my work laptop. it has kerberos

[jira] [Commented] (SPARK-29009) Returning pojo from udf not working

2019-09-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926592#comment-16926592 ] Hyukjin Kwon commented on SPARK-29009: -- Can you cope and paste of minimised version of the class to

[jira] [Commented] (SPARK-29038) SPIP: Support Spark Materialized View

2019-09-10 Thread Marco Gaido (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926650#comment-16926650 ] Marco Gaido commented on SPARK-29038: - [~cltlfcjin] currently spark has a something similar, which

[jira] [Comment Edited] (SPARK-29038) SPIP: Support Spark Materialized View

2019-09-10 Thread Marco Gaido (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926650#comment-16926650 ] Marco Gaido edited comment on SPARK-29038 at 9/10/19 1:40 PM: -- [~cltlfcjin]

[jira] [Updated] (SPARK-29029) PhysicalOperation.collectProjectsAndFilters should use AttributeMap while substituting aliases

2019-09-10 Thread Nikita Konda (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikita Konda updated SPARK-29029: - Component/s: SQL > PhysicalOperation.collectProjectsAndFilters should use AttributeMap while >

[jira] [Commented] (SPARK-29014) DataSourceV2: Clean up current, default, and session catalog uses

2019-09-10 Thread Ryan Blue (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926889#comment-16926889 ] Ryan Blue commented on SPARK-29014: --- [~cloud_fan], why does this require a major refactor? It would

[jira] [Created] (SPARK-29040) Support pyspark.createDataFrame from a pyarrow.Table

2019-09-10 Thread Bryan Cutler (Jira)
Bryan Cutler created SPARK-29040: Summary: Support pyspark.createDataFrame from a pyarrow.Table Key: SPARK-29040 URL: https://issues.apache.org/jira/browse/SPARK-29040 Project: Spark Issue

[jira] [Updated] (SPARK-29036) SparkThriftServer may can't cancel job after call a cancel before start.

2019-09-10 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-29036: -- Description: Disscuss in [https://github.com/apache/spark/pull/25611] > SparkThriftServer may can't

[jira] [Comment Edited] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-10 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926420#comment-16926420 ] Gabor Somogyi edited comment on SPARK-29027 at 9/10/19 1:38 PM:

[jira] [Comment Edited] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-10 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926577#comment-16926577 ] koert kuipers edited comment on SPARK-29027 at 9/10/19 1:02 PM: i am

[jira] [Updated] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-10 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29037: Description: For a stage, whose tasks commit output, a task saves output to a staging dir firstly, when

[jira] [Commented] (SPARK-29015) Can not support "add jar" on JDK 11

2019-09-10 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926502#comment-16926502 ] Yuming Wang commented on SPARK-29015: - Moved {{Case 2}} to SPARK-29022. It's another issue. > Can

[jira] [Updated] (SPARK-29015) Can not support "add jar" on JDK 11

2019-09-10 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-29015: Description: How to reproduce: {code:bash} export JAVA_HOME=/usr/lib/jdk-11.0.3 export

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-10 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926563#comment-16926563 ] koert kuipers commented on SPARK-29027: --- hey the command i run is: mvn clean test -fae i am not

[jira] [Resolved] (SPARK-28856) DataSourceV2: Support SHOW DATABASES

2019-09-10 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-28856. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25601

[jira] [Assigned] (SPARK-28856) DataSourceV2: Support SHOW DATABASES

2019-09-10 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-28856: --- Assignee: Terry Kim > DataSourceV2: Support SHOW DATABASES >

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-10 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926648#comment-16926648 ] Gabor Somogyi commented on SPARK-29027: --- {quote}where/how do you see that in reactor

[jira] [Updated] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-10 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29037: Summary: [Core] Spark gives duplicate result when an application was killed and rerun (was: [Core] Spark

[jira] [Updated] (SPARK-29037) [Core] Spark may duplicate results when an application aborted and rerun

2019-09-10 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29037: Description: Case: A spark application was be killed due to long-running. Then we re-run this

[jira] [Updated] (SPARK-29037) [Core] Spark gives duplicate result when an application aborted and rerun

2019-09-10 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29037: Summary: [Core] Spark gives duplicate result when an application aborted and rerun (was: [Core] Spark

[jira] [Updated] (SPARK-29015) Can not support "add jar" on JDK 11

2019-09-10 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-29015: Description: How to reproduce: Case 1: {code:bash} export JAVA_HOME=/usr/lib/jdk-11.0.3 export

[jira] [Commented] (SPARK-29009) Returning pojo from udf not working

2019-09-10 Thread Tomasz Belina (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926552#comment-16926552 ] Tomasz Belina commented on SPARK-29009: --- I've dig  a little dipper into source code and it looks

[jira] [Created] (SPARK-29038) SPIP: Support Spark Materialized View

2019-09-10 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-29038: -- Summary: SPIP: Support Spark Materialized View Key: SPARK-29038 URL: https://issues.apache.org/jira/browse/SPARK-29038 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-29037) [Core] Spark may duplicate results when an application aborted and rerun

2019-09-10 Thread feiwang (Jira)
feiwang created SPARK-29037: --- Summary: [Core] Spark may duplicate results when an application aborted and rerun Key: SPARK-29037 URL: https://issues.apache.org/jira/browse/SPARK-29037 Project: Spark

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-10 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926620#comment-16926620 ] koert kuipers commented on SPARK-29027: --- i am going to try running tests on a virtual machine to

[jira] [Commented] (SPARK-29009) Returning pojo from udf not working

2019-09-10 Thread Tomasz Belina (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926554#comment-16926554 ] Tomasz Belina commented on SPARK-29009: --- POJO is fine - I've just paste only part of the class and

[jira] [Commented] (SPARK-29038) SPIP: Support Spark Materialized View

2019-09-10 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926658#comment-16926658 ] angerszhu commented on SPARK-29038: --- I am doing a similar framework. It can trigger cache sub-query

[jira] [Updated] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-10 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29037: Affects Version/s: (was: 2.3.1) 2.1.0 > [Core] Spark gives duplicate result

[jira] [Resolved] (SPARK-28570) Shuffle Storage API: Use writer API in UnsafeShuffleWriter

2019-09-10 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28570. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25304

[jira] [Assigned] (SPARK-28570) Shuffle Storage API: Use writer API in UnsafeShuffleWriter

2019-09-10 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-28570: -- Assignee: Matt Cheah > Shuffle Storage API: Use writer API in UnsafeShuffleWriter >

[jira] [Updated] (SPARK-29045) Test failed due to table already exists in SQLMetricsSuite

2019-09-10 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-29045: --- Description: In method {{SQLMetricsTestUtils.testMetricsDynamicPartition()}}, there is a CREATE

[jira] [Created] (SPARK-29045) Test failed due to table already exists in SQLMetricsSuite

2019-09-10 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-29045: -- Summary: Test failed due to table already exists in SQLMetricsSuite Key: SPARK-29045 URL: https://issues.apache.org/jira/browse/SPARK-29045 Project: Spark Issue

[jira] [Commented] (SPARK-28902) Spark ML Pipeline with nested Pipelines fails to load when saved from Python

2019-09-10 Thread Saif Addin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927077#comment-16927077 ] Saif Addin commented on SPARK-28902: Ah, here I thought you said you couldn't reproduce it. Gladly

[jira] [Updated] (SPARK-29041) Allow createDataFrame to accept bytes as binary type

2019-09-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29041: - Description: {code} spark.createDataFrame([[b"abcd"]], "col binary") {code} simply fails.

[jira] [Updated] (SPARK-29041) Allow createDataFrame to accept bytes as binary type

2019-09-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29041: - Description: {code} spark.createDataFrame([[b"abcd"]], "col binary") {code} simply fails as

[jira] [Commented] (SPARK-29043) [History Server]Only one replay thread of FsHistoryProvider work because of straggler

2019-09-10 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927205#comment-16927205 ] Jungtaek Lim commented on SPARK-29043: -- It's asynchronous for replaying logs: it's synchronous for

[jira] [Commented] (SPARK-28902) Spark ML Pipeline with nested Pipelines fails to load when saved from Python

2019-09-10 Thread Junichi Koizumi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927076#comment-16927076 ] Junichi Koizumi commented on SPARK-28902: --- Since versions aren't the main concern here

[jira] [Updated] (SPARK-29001) Print better log when process of events becomes slow

2019-09-10 Thread Xingbo Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xingbo Jiang updated SPARK-29001: - Description: We shall print better log when process of events becomes slow, to help find out

[jira] [Updated] (SPARK-29001) Print better log when process of events becomes slow

2019-09-10 Thread Xingbo Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xingbo Jiang updated SPARK-29001: - Summary: Print better log when process of events becomes slow (was: Print event thread stack

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-10 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927189#comment-16927189 ] Jungtaek Lim commented on SPARK-29027: -- [~koert] Please try to mv krb5.conf to other and run the

[jira] [Comment Edited] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-10 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927189#comment-16927189 ] Jungtaek Lim edited comment on SPARK-29027 at 9/11/19 1:59 AM: --- [~koert]

[jira] [Updated] (SPARK-29043) [History Server]Only one replay thread of FsHistoryProvider work because of straggler

2019-09-10 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29043: Description: As shown in the attachment, we set spark.history.fs.numReplayThreads=30 for spark history

[jira] [Updated] (SPARK-29043) [History Server]Only one replay thread of FsHistoryProvider work because of straggler

2019-09-10 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29043: Description: As shown in the attachment, we set spark.history.fs.numReplayThreads=30 for spark history

[jira] [Updated] (SPARK-29043) [History Server]Only one replay thread of FsHistoryProvider work because of straggler

2019-09-10 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29043: Description: As shown in the attachment, we set spark.history.fs.numReplayThreads=30 for spark history

[jira] [Commented] (SPARK-29043) [History Server]Only one replay thread of FsHistoryProvider work because of straggler

2019-09-10 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927198#comment-16927198 ] feiwang commented on SPARK-29043: - I think we can change it to Asynchronous. > [History Server]Only

[jira] [Comment Edited] (SPARK-29043) [History Server]Only one replay thread of FsHistoryProvider work because of straggler

2019-09-10 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927198#comment-16927198 ] feiwang edited comment on SPARK-29043 at 9/11/19 2:26 AM: -- I think it is better

[jira] [Issue Comment Deleted] (SPARK-28902) Spark ML Pipeline with nested Pipelines fails to load when saved from Python

2019-09-10 Thread Junichi Koizumi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junichi Koizumi updated SPARK-28902: -- Comment: was deleted (was:   Since, versions aren't the main concern here should I

[jira] [Commented] (SPARK-28902) Spark ML Pipeline with nested Pipelines fails to load when saved from Python

2019-09-10 Thread Junichi Koizumi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927074#comment-16927074 ] Junichi Koizumi commented on SPARK-28902: ---   Since, versions aren't the main concern here

[jira] [Assigned] (SPARK-29026) Improve error message when constructor in `ScalaReflection` isn't found

2019-09-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-29026: Assignee: Mick Jermsurawong > Improve error message when constructor in

[jira] [Resolved] (SPARK-29026) Improve error message when constructor in `ScalaReflection` isn't found

2019-09-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29026. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25736

[jira] [Issue Comment Deleted] (SPARK-29038) SPIP: Support Spark Materialized View

2019-09-10 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-29038: -- Comment: was deleted (was: I am doing a similar framework. It can trigger cache sub-query data of

[jira] [Created] (SPARK-29043) [History Server]Only one replay thread of FsHistoryProvider work because of straggler

2019-09-10 Thread feiwang (Jira)
feiwang created SPARK-29043: --- Summary: [History Server]Only one replay thread of FsHistoryProvider work because of straggler Key: SPARK-29043 URL: https://issues.apache.org/jira/browse/SPARK-29043 Project:

[jira] [Created] (SPARK-29044) Resolved attribute(s) R#661751,residue#661752 missing from ipi#660814,residue#660731,exper_set#660827,R#660730,description#660815,sequence#660817,exper#660828,symbol#660

2019-09-10 Thread Kristine Senkane (Jira)
Kristine Senkane created SPARK-29044: Summary: Resolved attribute(s) R#661751,residue#661752 missing from ipi#660814,residue#660731,exper_set#660827,R#660730,description#660815,sequence#660817,exper#660828,symbol#660816 Key:

[jira] [Commented] (SPARK-29038) SPIP: Support Spark Materialized View

2019-09-10 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927217#comment-16927217 ] angerszhu commented on SPARK-29038: --- [~cltlfcjin]   *precalculating, alittle like CarbonData's Data

[jira] [Updated] (SPARK-29041) Allow createDataFrame to accept bytes as binary type

2019-09-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29041: - Description: {code} spark.createDataFrame([[b"abcd"]], "col binary") {code} simply fails as

[jira] [Resolved] (SPARK-25157) Streaming of image files from directory

2019-09-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25157. -- Resolution: Duplicate > Streaming of image files from directory >

[jira] [Updated] (SPARK-29043) [History Server]Only one replay thread of FsHistoryProvider work because of straggler

2019-09-10 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29043: Attachment: screenshot-1.png > [History Server]Only one replay thread of FsHistoryProvider work because

[jira] [Updated] (SPARK-29043) [History Server]Only one replay thread of FsHistoryProvider work because of straggler

2019-09-10 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29043: Description: As shown in the attachment, we set spark.history.fs.numReplayThreads=30 for spark history

[jira] [Updated] (SPARK-29043) [History Server]Only one replay thread of FsHistoryProvider work because of straggler

2019-09-10 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29043: Description: As shown in the attachment, we set spark.history.fs.numReplayThreads=30 for spark history

[jira] [Comment Edited] (SPARK-29038) SPIP: Support Spark Materialized View

2019-09-10 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927214#comment-16927214 ] Lantao Jin edited comment on SPARK-29038 at 9/11/19 3:24 AM: - [~mgaido]

[jira] [Created] (SPARK-29041) Allow createDataFrame to accept bytes as binary type

2019-09-10 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-29041: Summary: Allow createDataFrame to accept bytes as binary type Key: SPARK-29041 URL: https://issues.apache.org/jira/browse/SPARK-29041 Project: Spark Issue

[jira] [Created] (SPARK-29042) Sampling-based RDD with unordered input should be INDETERMINATE

2019-09-10 Thread Liang-Chi Hsieh (Jira)
Liang-Chi Hsieh created SPARK-29042: --- Summary: Sampling-based RDD with unordered input should be INDETERMINATE Key: SPARK-29042 URL: https://issues.apache.org/jira/browse/SPARK-29042 Project: Spark

[jira] [Commented] (SPARK-29038) SPIP: Support Spark Materialized View

2019-09-10 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927214#comment-16927214 ] Lantao Jin commented on SPARK-29038: [~mgaido] IIUC, there is no "query caching" in Spark, even no

[jira] [Updated] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-10 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29037: Description: For a stage, whose tasks commit output, a task saves output to a staging dir firstly, when

[jira] [Commented] (SPARK-28927) ArrayIndexOutOfBoundsException and Not-stable AUC metrics in ALS for datasets with 12 billion instances

2019-09-10 Thread Liang-Chi Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926814#comment-16926814 ] Liang-Chi Hsieh commented on SPARK-28927: - Hi [~JerryHouse], do you use any non-deterministic

[jira] [Resolved] (SPARK-28982) Support ThriftServer GetTypeInfoOperation for Spark's own type

2019-09-10 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-28982. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25694

[jira] [Assigned] (SPARK-28982) Support ThriftServer GetTypeInfoOperation for Spark's own type

2019-09-10 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-28982: --- Assignee: angerszhu > Support ThriftServer GetTypeInfoOperation for Spark's own type >

[jira] [Comment Edited] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-10 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926707#comment-16926707 ] koert kuipers edited comment on SPARK-29027 at 9/10/19 2:53 PM: i tried

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-10 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926707#comment-16926707 ] koert kuipers commented on SPARK-29027: --- i tried doing tests in a virtual machine and they pass so

[jira] [Assigned] (SPARK-29028) Add links to IBM Cloud Object Storage connector in cloud-integration.md

2019-09-10 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-29028: - Assignee: Dilip Biswal > Add links to IBM Cloud Object Storage connector in

[jira] [Resolved] (SPARK-29028) Add links to IBM Cloud Object Storage connector in cloud-integration.md

2019-09-10 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-29028. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25737

[jira] [Created] (SPARK-29039) centralize the catalog and table lookup logic

2019-09-10 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-29039: --- Summary: centralize the catalog and table lookup logic Key: SPARK-29039 URL: https://issues.apache.org/jira/browse/SPARK-29039 Project: Spark Issue Type:

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-10 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926809#comment-16926809 ] koert kuipers commented on SPARK-29027: --- [~gsomogyi] do you use any services that require open

[jira] [Updated] (SPARK-29024) Ignore case while resolving time zones

2019-09-10 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-29024: --- Summary: Ignore case while resolving time zones (was: Support the `zulu` time zone) > Ignore case

[jira] [Created] (SPARK-29031) Materialized column to accelerate queries

2019-09-10 Thread Jason Guo (Jira)
Jason Guo created SPARK-29031: - Summary: Materialized column to accelerate queries Key: SPARK-29031 URL: https://issues.apache.org/jira/browse/SPARK-29031 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-29006) Support special date/timestamp values `infinity`/`-infinity`

2019-09-10 Thread Anurag Sharma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926372#comment-16926372 ] Anurag Sharma commented on SPARK-29006: --- [~maxgekk] Thanks, will wait for your code to be merged. 

[jira] [Created] (SPARK-29032) Simplify Prometheus support by adding `PrometheusServlet`

2019-09-10 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-29032: - Summary: Simplify Prometheus support by adding `PrometheusServlet` Key: SPARK-29032 URL: https://issues.apache.org/jira/browse/SPARK-29032 Project: Spark

[jira] [Updated] (SPARK-29032) Simplify Prometheus support by adding PrometheusServlet

2019-09-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29032: -- Description: This issue aims to simplify `Prometheus` support in Spark standalone environment

[jira] [Updated] (SPARK-29033) Always use CreateNamedStructUnsafe codepath

2019-09-10 Thread Josh Rosen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-29033: --- Description: Spark 2.x has two separate implementations of the "create named struct" expression:

[jira] [Created] (SPARK-29034) String Constants with C-style Escapes

2019-09-10 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-29034: --- Summary: String Constants with C-style Escapes Key: SPARK-29034 URL: https://issues.apache.org/jira/browse/SPARK-29034 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-29031) Materialized column to accelerate queries

2019-09-10 Thread Jason Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Guo updated SPARK-29031: -- Description: Goals * Add a new SQL grammar of Materialized column * Implicitly rewrite SQL queries

[jira] [Created] (SPARK-29035) unpersist() ignoring cache/persist()

2019-09-10 Thread Jose Silva (Jira)
Jose Silva created SPARK-29035: -- Summary: unpersist() ignoring cache/persist() Key: SPARK-29035 URL: https://issues.apache.org/jira/browse/SPARK-29035 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-29032) Simplify Prometheus support by adding PrometheusServlet

2019-09-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29032: -- Summary: Simplify Prometheus support by adding PrometheusServlet (was: Simplify Prometheus

[jira] [Updated] (SPARK-29032) Simplify Prometheus support by adding PrometheusServlet`

2019-09-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29032: -- Summary: Simplify Prometheus support by adding PrometheusServlet` (was: Simplify Prometheus

  1   2   >