[jira] [Resolved] (SPARK-21733) ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

2017-08-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-21733. - Resolution: Not A Problem > ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM >

[jira] [Commented] (SPARK-21733) ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

2017-08-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136334#comment-16136334 ] Saisai Shao commented on SPARK-21733: - I'm going to close this issue again because the behavior is

[jira] [Comment Edited] (SPARK-21733) ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

2017-08-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136326#comment-16136326 ] Saisai Shao edited comment on SPARK-21733 at 8/22/17 5:41 AM: -- This is

[jira] [Commented] (SPARK-21733) ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

2017-08-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136326#comment-16136326 ] Saisai Shao commented on SPARK-21733: - This is because executor is killed by NM with SIGTERM, this is

[jira] [Commented] (SPARK-19109) ORC metadata section can sometimes exceed protobuf message size limit

2017-08-21 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136294#comment-16136294 ] Dongjoon Hyun commented on SPARK-19109: --- Thanks, [~wangchao2017], but I think there are

[jira] [Comment Edited] (SPARK-21733) ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

2017-08-21 Thread Jepson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136260#comment-16136260 ] Jepson edited comment on SPARK-21733 at 8/22/17 4:32 AM: - *The nodemanager log

[jira] [Comment Edited] (SPARK-21733) ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

2017-08-21 Thread Jepson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136260#comment-16136260 ] Jepson edited comment on SPARK-21733 at 8/22/17 4:29 AM: - *The nodemanager log

[jira] [Comment Edited] (SPARK-21802) Make sparkR MLP summary() expose probability column

2017-08-21 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136234#comment-16136234 ] Weichen Xu edited comment on SPARK-21802 at 8/22/17 4:25 AM: - cc

[jira] [Commented] (SPARK-21803) Remove the HiveDDLCommandSuite

2017-08-21 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136259#comment-16136259 ] Xiao Li commented on SPARK-21803: - https://github.com/apache/spark/pull/19015 > Remove the

[jira] [Commented] (SPARK-21733) ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

2017-08-21 Thread Jepson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136260#comment-16136260 ] Jepson commented on SPARK-21733: *The nodemanager log detail:* {code:java} 2017-08-22 11:20:07,984

[jira] [Created] (SPARK-21803) Remove the HiveDDLCommandSuite

2017-08-21 Thread Xiao Li (JIRA)
Xiao Li created SPARK-21803: --- Summary: Remove the HiveDDLCommandSuite Key: SPARK-21803 URL: https://issues.apache.org/jira/browse/SPARK-21803 Project: Spark Issue Type: Improvement

[jira] [Closed] (SPARK-21794) exception about reading task serial data(broadcast) value when the storage memory is not enough to unroll

2017-08-21 Thread roncenzhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao closed SPARK-21794. -- Resolution: Duplicate > exception about reading task serial data(broadcast) value when the storage >

[jira] [Commented] (SPARK-21794) exception about reading task serial data(broadcast) value when the storage memory is not enough to unroll

2017-08-21 Thread roncenzhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136237#comment-16136237 ] roncenzhao commented on SPARK-21794: [~yuming] Thanks, this is resolved by your PR. I close this

[jira] [Issue Comment Deleted] (SPARK-21733) ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

2017-08-21 Thread Jepson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jepson updated SPARK-21733: --- Comment: was deleted (was: I have resolve this issue.Thanks for [~jerryshao] and [~srowen] spark-submit \

[jira] [Issue Comment Deleted] (SPARK-21733) ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

2017-08-21 Thread Jepson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jepson updated SPARK-21733: --- Comment: was deleted (was: [~sowen], thank you for correcting me, I will notice it at the next time.) >

[jira] [Commented] (SPARK-21802) Make sparkR MLP summary() expose probability column

2017-08-21 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136234#comment-16136234 ] Weichen Xu commented on SPARK-21802: cc [~felixcheung] > Make sparkR MLP summary() expose

[jira] [Updated] (SPARK-21794) exception about reading task serial data(broadcast) value when the storage memory is not enough to unroll

2017-08-21 Thread roncenzhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-21794: --- Affects Version/s: (was: 2.1.1) 2.1.0 > exception about reading task

[jira] [Reopened] (SPARK-21733) ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

2017-08-21 Thread Jepson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jepson reopened SPARK-21733: *This error is happening again.* > ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM >

[jira] [Created] (SPARK-21802) Make sparkR MLP summary() expose probability column

2017-08-21 Thread Weichen Xu (JIRA)
Weichen Xu created SPARK-21802: -- Summary: Make sparkR MLP summary() expose probability column Key: SPARK-21802 URL: https://issues.apache.org/jira/browse/SPARK-21802 Project: Spark Issue Type:

[jira] [Created] (SPARK-21801) SparkR unit test randomly fail on trees

2017-08-21 Thread Weichen Xu (JIRA)
Weichen Xu created SPARK-21801: -- Summary: SparkR unit test randomly fail on trees Key: SPARK-21801 URL: https://issues.apache.org/jira/browse/SPARK-21801 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-21801) SparkR unit test randomly fail on trees

2017-08-21 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136229#comment-16136229 ] Weichen Xu commented on SPARK-21801: cc [~felixcheung] Can you help fix this ? > SparkR unit test

[jira] [Commented] (SPARK-21798) No config to replace deprecated SPARK_CLASSPATH config for launching daemons like History Server

2017-08-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136196#comment-16136196 ] Saisai Shao commented on SPARK-21798: - I think this one could be used, looks like there's no other

[jira] [Closed] (SPARK-21796) pyspark count failed in python3.5.2

2017-08-21 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cen yuhai closed SPARK-21796. - Resolution: Not A Problem > pyspark count failed in python3.5.2 > --- >

[jira] [Commented] (SPARK-21796) pyspark count failed in python3.5.2

2017-08-21 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136189#comment-16136189 ] cen yuhai commented on SPARK-21796: --- It is env problem, I will reinstall system, close this issue. >

[jira] [Resolved] (SPARK-21753) running pi example with pypy on spark fails to serialize

2017-08-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-21753. -- Resolution: Fixed Assignee: Kyle Kelley Fix Version/s: 2.3.0 Fixed in

[jira] [Commented] (SPARK-21753) running pi example with pypy on spark fails to serialize

2017-08-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136180#comment-16136180 ] Hyukjin Kwon commented on SPARK-21753: -- I merged her PR and double checked if it works: Before:

[jira] [Commented] (SPARK-21799) KMeans performance regression (5-6x slowdown) in Spark 2.2

2017-08-21 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136175#comment-16136175 ] Liang-Chi Hsieh commented on SPARK-21799: - So I think the problem is you shouldn't do

[jira] [Resolved] (SPARK-19690) Join a streaming DataFrame with a batch DataFrame may not work

2017-08-21 Thread Jose Torres (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jose Torres resolved SPARK-19690. - Resolution: Duplicate SPARK-21765 > Join a streaming DataFrame with a batch DataFrame may not

[jira] [Commented] (SPARK-19690) Join a streaming DataFrame with a batch DataFrame may not work

2017-08-21 Thread Jose Torres (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136171#comment-16136171 ] Jose Torres commented on SPARK-19690: - This will be fixed by SPARK-21765; we can restrict the

[jira] [Issue Comment Deleted] (SPARK-21799) KMeans performance regression (5-6x slowdown) in Spark 2.2

2017-08-21 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-21799: Comment: was deleted (was: Yeah, that looks right direction. {{df.storageLevel}} is not

[jira] [Assigned] (SPARK-21070) Pick up cloudpickle upgrades from cloudpickle python module

2017-08-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-21070: Assignee: Kyle Kelley > Pick up cloudpickle upgrades from cloudpickle python module >

[jira] [Commented] (SPARK-21799) KMeans performance regression (5-6x slowdown) in Spark 2.2

2017-08-21 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136164#comment-16136164 ] Liang-Chi Hsieh commented on SPARK-21799: - Hmm, I go to check ML KMeans codes where I don't find

[jira] [Resolved] (SPARK-21070) Pick up cloudpickle upgrades from cloudpickle python module

2017-08-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-21070. -- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18734

[jira] [Commented] (SPARK-19109) ORC metadata section can sometimes exceed protobuf message size limit

2017-08-21 Thread sydt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136154#comment-16136154 ] sydt commented on SPARK-19109: -- Thanks for your reply. I have compiled hive-exec-1.2.1-spark2.jar and it is

[jira] [Commented] (SPARK-21799) KMeans performance regression (5-6x slowdown) in Spark 2.2

2017-08-21 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136152#comment-16136152 ] Liang-Chi Hsieh commented on SPARK-21799: - Yeah, that looks right direction. {{df.storageLevel}}

[jira] [Resolved] (SPARK-21617) ALTER TABLE...ADD COLUMNS broken in Hive 2.1 for DS tables

2017-08-21 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-21617. - Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.3.0 2.2.1

[jira] [Commented] (SPARK-21800) java.lang.reflect.InvocationTargetException: java.lang.ClassCastException........ Spark is creating the LocalRelation when using Save as Hive table which is th

2017-08-21 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135868#comment-16135868 ] Sean Owen commented on SPARK-21800: --- This doesn't show the actual ClassCastException and isn't

[jira] [Reopened] (SPARK-17742) Spark Launcher does not get failed state in Listener

2017-08-21 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reopened SPARK-17742: Assignee: Marcelo Vanzin Reopening temporarily for a follow up fix:

[jira] [Updated] (SPARK-21800) java.lang.reflect.InvocationTargetException: java.lang.ClassCastException........ Spark is creating the LocalRelation when using Save as Hive table which is thro

2017-08-21 Thread NAVEEN RAJ KURAPATI (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] NAVEEN RAJ KURAPATI updated SPARK-21800: Component/s: SQL > java.lang.reflect.InvocationTargetException: >

[jira] [Created] (SPARK-21800) java.lang.reflect.InvocationTargetException: java.lang.ClassCastException........ Spark is creating the LocalRelation when using Save as Hive table which is thro

2017-08-21 Thread NAVEEN RAJ KURAPATI (JIRA)
NAVEEN RAJ KURAPATI created SPARK-21800: --- Summary: java.lang.reflect.InvocationTargetException: java.lang.ClassCastException Spark is creating the LocalRelation when using Save as Hive table which is throwing classcast exception

[jira] [Commented] (SPARK-19552) Upgrade Netty version to 4.1.8 final

2017-08-21 Thread Charles Allen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135819#comment-16135819 ] Charles Allen commented on SPARK-19552: --- [~aash] Do you have a link to an Apache Arrow issue on

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

2017-08-21 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135669#comment-16135669 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/21/17 8:01 PM:

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

2017-08-21 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135669#comment-16135669 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/21/17 7:58 PM:

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

2017-08-21 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135669#comment-16135669 ] Stavros Kontopoulos commented on SPARK-21752: - In order to be correct before we update

[jira] [Commented] (SPARK-21549) Spark fails to complete job correctly in case of OutputFormat which do not write into hdfs

2017-08-21 Thread Sergey Zhemzhitsky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135545#comment-16135545 ] Sergey Zhemzhitsky commented on SPARK-21549: [~mridulm80], [~WeiqingYang] does it make sense

[jira] [Updated] (SPARK-21799) KMeans performance regression (5-6x slowdown) in Spark 2.2

2017-08-21 Thread Siddharth Murching (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Murching updated SPARK-21799: --- Description: I've been running KMeans performance tests using

[jira] [Updated] (SPARK-21799) KMeans performance regression (5-6x slowdown) in Spark 2.2

2017-08-21 Thread Siddharth Murching (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Murching updated SPARK-21799: --- Description: I've been running KMeans performance tests using

[jira] [Updated] (SPARK-21799) KMeans performance regression (5-6x slowdown) in Spark 2.2

2017-08-21 Thread Siddharth Murching (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Murching updated SPARK-21799: --- Summary: KMeans performance regression (5-6x slowdown) in Spark 2.2 (was: KMeans

[jira] [Updated] (SPARK-21799) KMeans Performance Regression (5-6x slowdown) in Spark 2.2

2017-08-21 Thread Siddharth Murching (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Murching updated SPARK-21799: --- Description: I've been running KMeans performance tests using

[jira] [Created] (SPARK-21799) KMeans Performance Regression (5-6x slowdown) in Spark 2.2

2017-08-21 Thread Siddharth Murching (JIRA)
Siddharth Murching created SPARK-21799: -- Summary: KMeans Performance Regression (5-6x slowdown) in Spark 2.2 Key: SPARK-21799 URL: https://issues.apache.org/jira/browse/SPARK-21799 Project: Spark

[jira] [Commented] (SPARK-19109) ORC metadata section can sometimes exceed protobuf message size limit

2017-08-21 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135422#comment-16135422 ] Dongjoon Hyun commented on SPARK-19109: --- Thanks, Nic Eggert and sydt. I see. I just wanted to

[jira] [Comment Edited] (SPARK-19109) ORC metadata section can sometimes exceed protobuf message size limit

2017-08-21 Thread Nic Eggert (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135241#comment-16135241 ] Nic Eggert edited comment on SPARK-19109 at 8/21/17 2:43 PM: - [~dongjoon] I

[jira] [Commented] (SPARK-19109) ORC metadata section can sometimes exceed protobuf message size limit

2017-08-21 Thread Nic Eggert (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135241#comment-16135241 ] Nic Eggert commented on SPARK-19109: [~dongjoon] I don't have the ability to reproduce it on my end

[jira] [Commented] (SPARK-21797) spark cannot read partitioned data in S3 that are partly in glacier

2017-08-21 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135236#comment-16135236 ] Sean Owen commented on SPARK-21797: --- Sure, but this would not be a change in Spark, but in the AWS SDK

[jira] [Updated] (SPARK-21798) No config to replace deprecated SPARK_CLASSPATH config for launching daemons like History Server

2017-08-21 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21798: -- Issue Type: Improvement (was: Bug) > No config to replace deprecated SPARK_CLASSPATH config for

[jira] [Created] (SPARK-21798) No config to replace deprecated SPARK_CLASSPATH config for launching daemons like History Server

2017-08-21 Thread Sanket Reddy (JIRA)
Sanket Reddy created SPARK-21798: Summary: No config to replace deprecated SPARK_CLASSPATH config for launching daemons like History Server Key: SPARK-21798 URL: https://issues.apache.org/jira/browse/SPARK-21798

[jira] [Commented] (SPARK-21797) spark cannot read partitioned data in S3 that are partly in glacier

2017-08-21 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-21797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135233#comment-16135233 ] Boris Clémençon commented on SPARK-21797: -- Hi Sean, Thanks for the quick answer. I understand

[jira] [Resolved] (SPARK-21797) spark cannot read partitioned data in S3 that are partly in glacier

2017-08-21 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21797. --- Resolution: Not A Problem This sounds like a question about how Amazon's SDK and Glacier work, not a

[jira] [Updated] (SPARK-21797) spark cannot read partitioned data in S3 that are partly in glacier

2017-08-21 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-21797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Clémençon updated SPARK-21797: - Priority: Major (was: Critical) > spark cannot read partitioned data in S3 that are

[jira] [Updated] (SPARK-21797) spark cannot read partitioned data in S3 that are partly in glacier

2017-08-21 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-21797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Clémençon updated SPARK-21797: - Description: I have a dataset in parquet in S3 partitioned by date (dt) with oldest date

[jira] [Updated] (SPARK-21797) spark cannot read partitioned data in S3 that are partly in glacier

2017-08-21 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-21797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Clémençon updated SPARK-21797: - Description: I have a dataset in parquet in S3 partitioned by date (dt) with oldest date

[jira] [Created] (SPARK-21797) spark cannot read partitioned data in S3 that are partly in glacier

2017-08-21 Thread JIRA
Boris Clémençon created SPARK-21797: Summary: spark cannot read partitioned data in S3 that are partly in glacier Key: SPARK-21797 URL: https://issues.apache.org/jira/browse/SPARK-21797 Project:

[jira] [Assigned] (SPARK-21468) FeatureHasher Python API

2017-08-21 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath reassigned SPARK-21468: -- Assignee: Nick Pentreath > FeatureHasher Python API > > >

[jira] [Resolved] (SPARK-21468) FeatureHasher Python API

2017-08-21 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath resolved SPARK-21468. Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18970

[jira] [Resolved] (SPARK-21718) Heavy log of type: "Skipping partition based on stats ..."

2017-08-21 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-21718. --- Resolution: Fixed Fix Version/s: 2.3.0 > Heavy log of type: "Skipping

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

2017-08-21 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135082#comment-16135082 ] Sean Owen commented on SPARK-21752: --- Docs can't hurt, especially if they can be applied consistently. I

[jira] [Commented] (SPARK-14540) Support Scala 2.12 closures and Java 8 lambdas in ClosureCleaner

2017-08-21 Thread Roman Iakovlev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135053#comment-16135053 ] Roman Iakovlev commented on SPARK-14540: This issue looks like one of the biggest obstacles for

[jira] [Commented] (SPARK-21796) pyspark count failed in python3.5.2

2017-08-21 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135048#comment-16135048 ] cen yuhai commented on SPARK-21796: --- I can execute the code by spark-submit or zeppelin > pyspark

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

2017-08-21 Thread Jakub Nowacki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135015#comment-16135015 ] Jakub Nowacki commented on SPARK-21752: --- [~srowen] Do you think we can create some sort of

[jira] [Commented] (SPARK-21787) Support for pushing down filters for date types in ORC

2017-08-21 Thread Stefan de Koning (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135005#comment-16135005 ] Stefan de Koning commented on SPARK-21787: -- Thanks! > Support for pushing down filters for date

[jira] [Commented] (SPARK-21796) pyspark count failed in python3.5.2

2017-08-21 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134988#comment-16134988 ] cen yuhai commented on SPARK-21796: --- !screenshot-1.png! This is my script, I just change the version,

[jira] [Updated] (SPARK-21796) pyspark count failed in python3.5.2

2017-08-21 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cen yuhai updated SPARK-21796: -- Attachment: screenshot-1.png > pyspark count failed in python3.5.2 >

[jira] [Commented] (SPARK-21796) pyspark count failed in python3.5.2

2017-08-21 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134960#comment-16134960 ] Sean Owen commented on SPARK-21796: --- OK, but you still likely have some problem if your Python

[jira] [Comment Edited] (SPARK-21796) pyspark count failed in python3.5.2

2017-08-21 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134958#comment-16134958 ] cen yuhai edited comment on SPARK-21796 at 8/21/17 9:44 AM: [~srowen] hi

[jira] [Comment Edited] (SPARK-21796) pyspark count failed in python3.5.2

2017-08-21 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134958#comment-16134958 ] cen yuhai edited comment on SPARK-21796 at 8/21/17 9:44 AM: [~srowen] hi

[jira] [Commented] (SPARK-21796) pyspark count failed in python3.5.2

2017-08-21 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134958#comment-16134958 ] cen yuhai commented on SPARK-21796: --- [~srowen] hi owean, I already set in spark-env.sh export

[jira] [Commented] (SPARK-21796) pyspark count failed in python3.5.2

2017-08-21 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134953#comment-16134953 ] Sean Owen commented on SPARK-21796: --- That pretty much demonstrates you have some Python env problem,

[jira] [Commented] (SPARK-21796) pyspark count failed in python3.5.2

2017-08-21 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134951#comment-16134951 ] cen yuhai commented on SPARK-21796: --- [~hyukjin.kwon] I upload the file. Some machines will failed,

[jira] [Updated] (SPARK-21796) pyspark count failed in python3.5.2

2017-08-21 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cen yuhai updated SPARK-21796: -- Attachment: user > pyspark count failed in python3.5.2 > --- > >

[jira] [Commented] (SPARK-21796) pyspark count failed in python3.5.2

2017-08-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134949#comment-16134949 ] Hyukjin Kwon commented on SPARK-21796: -- Could you share your input file? I can't reproduce in the

[jira] [Comment Edited] (SPARK-13330) PYTHONHASHSEED is not propgated to python worker

2017-08-21 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134917#comment-16134917 ] cen yuhai edited comment on SPARK-13330 at 8/21/17 9:00 AM: [~zjffdu]

[jira] [Commented] (SPARK-13330) PYTHONHASHSEED is not propgated to python worker

2017-08-21 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134917#comment-16134917 ] cen yuhai commented on SPARK-13330: --- [~zjffdu] can you also look at this issue:

[jira] [Updated] (SPARK-21796) pyspark count failed in python3.5.2

2017-08-21 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cen yuhai updated SPARK-21796: -- Environment: Spark 2.1.1 Python 3.5.2 Anaconda3 4.2.0 (was: spark 2.1.1 Python 3.5.2 anaconda3

[jira] [Updated] (SPARK-21796) pyspark count failed in python3.5.2

2017-08-21 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cen yuhai updated SPARK-21796: -- Environment: Python 3.5.2 Anaconda3 4.2.0 (was: Spark 2.1.1 Python 3.5.2 Anaconda3 4.2.0) >

[jira] [Updated] (SPARK-21796) pyspark count failed in python3.5.2

2017-08-21 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cen yuhai updated SPARK-21796: -- Description: steps: {code} pyspark user_data =

[jira] [Created] (SPARK-21796) pyspark count failed in python3.5.2

2017-08-21 Thread cen yuhai (JIRA)
cen yuhai created SPARK-21796: - Summary: pyspark count failed in python3.5.2 Key: SPARK-21796 URL: https://issues.apache.org/jira/browse/SPARK-21796 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-19528) external shuffle service registration timeout is very short with heavy workloads when dynamic allocation is enabled

2017-08-21 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-19528: -- Target Version/s: (was: 1.6.2, 1.6.3) > external shuffle service registration timeout is very short

[jira] [Resolved] (SPARK-21775) Dynamic Log Level Settings for executors

2017-08-21 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21775. --- Resolution: Won't Fix > Dynamic Log Level Settings for executors >

[jira] [Assigned] (SPARK-21782) Repartition creates skews when numPartitions is a power of 2

2017-08-21 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-21782: - Assignee: Sergey Serebryakov > Repartition creates skews when numPartitions is a power of 2 >

[jira] [Resolved] (SPARK-21782) Repartition creates skews when numPartitions is a power of 2

2017-08-21 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21782. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18990

[jira] [Assigned] (SPARK-21718) Heavy log of type: "Skipping partition based on stats ..."

2017-08-21 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-21718: - Assignee: Sean Owen > Heavy log of type: "Skipping partition based on stats ..." >

[jira] [Updated] (SPARK-19109) ORC metadata section can sometimes exceed protobuf message size limit

2017-08-21 Thread sydt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sydt updated SPARK-19109: - Attachment: InsertPic_.png hi, I meet this problem and resolved by recompile source code of

[jira] [Commented] (SPARK-21794) exception about reading task serial data(broadcast) value when the storage memory is not enough to unroll

2017-08-21 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134752#comment-16134752 ] Yuming Wang commented on SPARK-21794: - Seems a duplicate of