[jira] [Issue Comment Deleted] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-18673: -- Comment: was deleted (was: I'm OOO, please expect slow email response, sorry for the inconvenience. ) >

[jira] [Commented] (SPARK-18107) Insert overwrite statement runs much slower in spark-sql than it does in hive-client

[ https://issues.apache.org/jira/browse/SPARK-18107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841351#comment-16841351 ] KaiXu commented on SPARK-18107: --- it seems this issue have not been fixed? I encountered this issue with

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823462#comment-16823462 ] KaiXu commented on SPARK-18673: --- I'm OOO, please expect slow email response, sorry for the inconvenience.

[jira] [Commented] (SPARK-27289) spark-submit explicit configuration does not take effect but Spark UI shows it's effective

[ https://issues.apache.org/jira/browse/SPARK-27289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814094#comment-16814094 ] KaiXu commented on SPARK-27289: --- I have verified that the intermediate data is written to spark.local.dir

[jira] [Commented] (SPARK-27289) spark-submit explicit configuration does not take effect but Spark UI shows it's effective

[ https://issues.apache.org/jira/browse/SPARK-27289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812103#comment-16812103 ] KaiXu commented on SPARK-27289: --- Do you check where the intermediate shuffle data was wrote while changing

[jira] [Updated] (SPARK-27289) spark-submit explicit configuration does not take effect but Spark UI shows it's effective

[ https://issues.apache.org/jira/browse/SPARK-27289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-27289: -- Attachment: Capture.PNG > spark-submit explicit configuration does not take effect but Spark UI shows > it's

[jira] [Created] (SPARK-27289) spark-submit explicit configuration does not take effect but Spark UI shows it's effective

KaiXu created SPARK-27289: - Summary: spark-submit explicit configuration does not take effect but Spark UI shows it's effective Key: SPARK-27289 URL: https://issues.apache.org/jira/browse/SPARK-27289

[jira] [Commented] (SPARK-27100) dag-scheduler-event-loop" java.lang.StackOverflowError

[ https://issues.apache.org/jira/browse/SPARK-27100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16793233#comment-16793233 ] KaiXu commented on SPARK-27100: --- hi [~hyukjin.kwon], the workload I'm running is ALS from Hibench, the

[jira] [Updated] (SPARK-27100) dag-scheduler-event-loop" java.lang.StackOverflowError

[ https://issues.apache.org/jira/browse/SPARK-27100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-27100: -- Attachment: stderr > dag-scheduler-event-loop" java.lang.StackOverflowError >

[jira] [Commented] (SPARK-27100) dag-scheduler-event-loop" java.lang.StackOverflowError

[ https://issues.apache.org/jira/browse/SPARK-27100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787620#comment-16787620 ] KaiXu commented on SPARK-27100: --- I have checked stderr log file(50M:() of the task,  the stack trace is

[jira] [Updated] (SPARK-27100) dag-scheduler-event-loop" java.lang.StackOverflowError

[ https://issues.apache.org/jira/browse/SPARK-27100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-27100: -- Affects Version/s: 2.3.3 > dag-scheduler-event-loop" java.lang.StackOverflowError >

[jira] [Commented] (SPARK-27100) dag-scheduler-event-loop" java.lang.StackOverflowError

[ https://issues.apache.org/jira/browse/SPARK-27100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787565#comment-16787565 ] KaiXu commented on SPARK-27100: --- Hi, [~yumwang], I tried Spark2.3.3, it also has the similar issue.

[jira] [Created] (SPARK-27100) dag-scheduler-event-loop" java.lang.StackOverflowError

KaiXu created SPARK-27100: - Summary: dag-scheduler-event-loop" java.lang.StackOverflowError Key: SPARK-27100 URL: https://issues.apache.org/jira/browse/SPARK-27100 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-19528) external shuffle service registration timeout is very short with heavy workloads when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-19528: -- Attachment: SPARK-19528.1.spark2.patch > external shuffle service registration timeout is very short with

[jira] [Commented] (SPARK-19528) external shuffle service registration timeout is very short with heavy workloads when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410655#comment-16410655 ] KaiXu commented on SPARK-19528: --- spark2.0.2 also found this issue recently! > external shuffle service

[jira] [Updated] (SPARK-19528) external shuffle service registration timeout is very short with heavy workloads when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-19528: -- Affects Version/s: 2.0.2 > external shuffle service registration timeout is very short with heavy > workloads

[jira] [Comment Edited] (SPARK-19528) external shuffle service registration timeout is very short with heavy workloads when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134305#comment-16134305 ] KaiXu edited comment on SPARK-19528 at 8/20/17 5:51 AM: Currently the shuffle

[jira] [Updated] (SPARK-19528) external shuffle service registration timeout is very short with heavy workloads when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-19528: -- Target Version/s: 1.6.3, 1.6.2 Summary: external shuffle service registration timeout is very

[jira] [Updated] (SPARK-19528) external shuffle service would close while still have request from executor when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-19528: -- Attachment: SPARK-19528.1.patch Currently the shuffle service registration timeout is 5s which is hardcoded.

[jira] [Updated] (SPARK-19528) external shuffle service would close while still have request from executor when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-19528: -- Affects Version/s: 1.6.3 > external shuffle service would close while still have request from executor > when

[jira] [Updated] (SPARK-19725) different parquet dependency in spark2.0.x and Hive2.x cause failure of HoS when using parquet file format

[ https://issues.apache.org/jira/browse/SPARK-19725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-19725: -- Description: the parquet version in hive2.x is 1.8.1 while in spark2.0.x is 1.7.0, so when run HoS queries

[jira] [Commented] (SPARK-19725) different parquet dependency in spark2.x and Hive2.x cause failure of HoS when using parquet file format

[ https://issues.apache.org/jira/browse/SPARK-19725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884208#comment-15884208 ] KaiXu commented on SPARK-19725: --- Hive supports spark2.x through HIVE-14029, spark supports Hive

[jira] [Created] (SPARK-19725) different parquet dependency in spark2.x and Hive2.x cause failure of HoS when using parquet file format

KaiXu created SPARK-19725: - Summary: different parquet dependency in spark2.x and Hive2.x cause failure of HoS when using parquet file format Key: SPARK-19725 URL: https://issues.apache.org/jira/browse/SPARK-19725

[jira] [Commented] (SPARK-19725) different parquet dependency in spark2.x and Hive2.x cause failure of HoS when using parquet file format

[ https://issues.apache.org/jira/browse/SPARK-19725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882311#comment-15882311 ] KaiXu commented on SPARK-19725: --- using parquet-provided profile can workaround this issue, but it's better

[jira] [Commented] (SPARK-19528) external shuffle service would close while still have request from executor when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865010#comment-15865010 ] KaiXu commented on SPARK-19528: --- nodemanager log see below exception, is that helpful? 2017-02-09

[jira] [Commented] (SPARK-19528) external shuffle service would close while still have request from executor when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864844#comment-15864844 ] KaiXu commented on SPARK-19528: --- Thanks [~zsxwing] for the comment, this issue can be occurred frequently,

[jira] [Updated] (SPARK-19569) could not get APP ID and cause failed to connect to spark driver on yarn-client mode

[ https://issues.apache.org/jira/browse/SPARK-19569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-19569: -- Description: when I run Hive queries on Spark, got below error in the console, after check the container's

[jira] [Updated] (SPARK-19569) could not get APP ID and cause failed to connect to spark driver on yarn-client mode

[ https://issues.apache.org/jira/browse/SPARK-19569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-19569: -- Description: when I run Hive queries on Spark, got below error in the console, after check the container's

[jira] [Updated] (SPARK-19569) could not get APP ID and failed to connect to spark driver on yarn-client mode

[ https://issues.apache.org/jira/browse/SPARK-19569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-19569: -- Summary: could not get APP ID and failed to connect to spark driver on yarn-client mode (was: could not

[jira] [Updated] (SPARK-19569) could not get APP ID and cause failed to connect to spark driver on yarn-client mode

[ https://issues.apache.org/jira/browse/SPARK-19569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-19569: -- Summary: could not get APP ID and cause failed to connect to spark driver on yarn-client mode (was: could

[jira] [Updated] (SPARK-19569) could not connect to spark driver on yarn-client mode

[ https://issues.apache.org/jira/browse/SPARK-19569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-19569: -- Description: when I run Hive queries on Spark, got below error in the console, after check the container's

[jira] [Commented] (SPARK-19569) could not connect to spark driver on yarn-client mode

[ https://issues.apache.org/jira/browse/SPARK-19569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863109#comment-15863109 ] KaiXu commented on SPARK-19569: --- it's not the IP address resolution issue (SPARK-5113), since 192.168.1.1

[jira] [Updated] (SPARK-19569) could not connect to spark driver on yarn-client mode

[ https://issues.apache.org/jira/browse/SPARK-19569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-19569: -- Description: when I run Hive queries on Spark, got below error in the console, after check the container's

[jira] [Created] (SPARK-19569) could not connect to spark driver on yarn-client mode

KaiXu created SPARK-19569: - Summary: could not connect to spark driver on yarn-client mode Key: SPARK-19569 URL: https://issues.apache.org/jira/browse/SPARK-19569 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-19528) external shuffle service would close while still have request from executor when dynamic allocation is enabled

KaiXu created SPARK-19528: - Summary: external shuffle service would close while still have request from executor when dynamic allocation is enabled Key: SPARK-19528 URL: https://issues.apache.org/jira/browse/SPARK-19528

[jira] [Commented] (SPARK-18443) spark leak memeory and led to OOM

[ https://issues.apache.org/jira/browse/SPARK-18443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15666550#comment-15666550 ] KaiXu commented on SPARK-18443: --- similar to this issue(https://issues.apache.org/jira/browse/SPARK-18289),

[jira] [Updated] (SPARK-18443) spark leak memeory and led to OOM

[ https://issues.apache.org/jira/browse/SPARK-18443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-18443: -- Environment: CentOS7.2 kernel: 3.10.0-327.el7.x86_64 Hadoop2.7.1 Spark1.6.2 release version Intel(R) Xeon(R)

[jira] [Created] (SPARK-18443) spark leak memeory and led to OOM

KaiXu created SPARK-18443: - Summary: spark leak memeory and led to OOM Key: SPARK-18443 URL: https://issues.apache.org/jira/browse/SPARK-18443 Project: Spark Issue Type: Bug Components:

[jira] [Commented] (SPARK-14560) Cooperative Memory Management for Spillables

[ https://issues.apache.org/jira/browse/SPARK-14560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15666005#comment-15666005 ] KaiXu commented on SPARK-14560: --- spark1.6.2 has the same issue: 16/11/15 10:11:15 INFO

[jira] [Commented] (SPARK-18289) spark.util.collection.ExternalSorter leak memory when task force spilling in-memory map to disk

[ https://issues.apache.org/jira/browse/SPARK-18289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15640994#comment-15640994 ] KaiXu commented on SPARK-18289: --- this issue can be reproduced with the above parameters. >

[jira] [Updated] (SPARK-18289) spark.util.collection.ExternalSorter leak memory when task force spilling in-memory map to disk

[ https://issues.apache.org/jira/browse/SPARK-18289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-18289: -- Description: We use BigBench to test the performance of Hive on Spark2.0 on Intel(R) Xeon(R) CPU E5-2699 v4(1

[jira] [Commented] (SPARK-18289) spark.util.collection.ExternalSorter leak memory when task force spilling in-memory map to disk

[ https://issues.apache.org/jira/browse/SPARK-18289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15639638#comment-15639638 ] KaiXu commented on SPARK-18289: --- find a similar issue:https://issues.apache.org/jira/browse/SPARK-11293 >

[jira] [Updated] (SPARK-18289) spark.util.collection.ExternalSorter leak memory when task force spilling in-memory map to disk

[ https://issues.apache.org/jira/browse/SPARK-18289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-18289: -- Description: We use BigBench to test the performance of Hive on Spark2.0 on Intel(R) Xeon(R) CPU E5-2699 v4(1

[jira] [Updated] (SPARK-18289) spark.util.collection.ExternalSorter leak memory when task force spilling in-memory map to disk

[ https://issues.apache.org/jira/browse/SPARK-18289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-18289: -- Description: We use BigBench to test the performance of Hive on Spark2.0 on Intel(R) Xeon(R) CPU E5-2699 v4(1

[jira] [Updated] (SPARK-18289) spark.util.collection.ExternalSorter leak memory when task force spilling in-memory map to disk

[ https://issues.apache.org/jira/browse/SPARK-18289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-18289: -- Description: We use BigBench to test the performance of Hive on Spark2.0 on Intel(R) Xeon(R) CPU E5-2699 v4(1

[jira] [Updated] (SPARK-18289) spark.util.collection.ExternalSorter leak memory when task force spilling in-memory map to disk

[ https://issues.apache.org/jira/browse/SPARK-18289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-18289: -- Description: We use BigBench to test the performance of Hive on Spark2.0 on Intel(R) Xeon(R) CPU E5-2699 v4(1

[jira] [Updated] (SPARK-18289) spark.util.collection.ExternalSorter leak memory when task force spilling in-memory map to disk

[ https://issues.apache.org/jira/browse/SPARK-18289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-18289: -- Description: We use BigBench to test the performance of Hive on Spark2.0 on Intel(R) Xeon(R) CPU E5-2699 v4(1

[jira] [Updated] (SPARK-18289) spark.util.collection.ExternalSorter leak memory when task force spilling in-memory map to disk

[ https://issues.apache.org/jira/browse/SPARK-18289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-18289: -- Description: We use BigBench to test the performance of Hive on Spark2.0 on Intel(R) Xeon(R) CPU E5-2699 v4(1

[jira] [Created] (SPARK-18289) spark.util.collection.ExternalSorter leak memory when task force spilling in-memory map to disk

KaiXu created SPARK-18289: - Summary: spark.util.collection.ExternalSorter leak memory when task force spilling in-memory map to disk Key: SPARK-18289 URL: https://issues.apache.org/jira/browse/SPARK-18289

[jira] [Updated] (SPARK-18289) spark.util.collection.ExternalSorter leak memory when task force spilling in-memory map to disk

[ https://issues.apache.org/jira/browse/SPARK-18289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-18289: -- Environment: CentOS7.2 kernel: 3.10.0-327.el7.x86_64 Hadoop2.7.1 Spark2.0.0 release version Hive2.1 with patch

[jira] [Updated] (SPARK-18289) spark.util.collection.ExternalSorter leak memory when task force spilling in-memory map to disk

[ https://issues.apache.org/jira/browse/SPARK-18289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-18289: -- Environment: CentOS7.2 kernel: 3.10.0-327.el7.x86_64 Hadoop2.7.1 Spark2.0.0 release version Hive2.1 with

[jira] [Commented] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610286#comment-15610286 ] KaiXu commented on SPARK-18112: --- Spark 2.0 removes JavaSparkListener and change SparkListener from a trait

[jira] [Created] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

KaiXu created SPARK-18112: - Summary: Spark2.x does not support read data from Hive 2.x metastore Key: SPARK-18112 URL: https://issues.apache.org/jira/browse/SPARK-18112 Project: Spark Issue Type: