[jira] [Commented] (SPARK-8417) spark-class has illegal statement

[ https://issues.apache.org/jira/browse/SPARK-8417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599899#comment-14599899 ] Kan Zhang commented on SPARK-8417: -- [~blipe] how to reproduce the error you saw? The

[jira] [Commented] (SPARK-8129) Securely pass auth secrets to executors in standalone cluster mode

[ https://issues.apache.org/jira/browse/SPARK-8129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14592243#comment-14592243 ] Kan Zhang commented on SPARK-8129: -- Checked on my Linux box that env variables are only

[jira] [Updated] (SPARK-8129) Securely pass auth secrets to executors in standalone cluster mode

[ https://issues.apache.org/jira/browse/SPARK-8129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated SPARK-8129: - Description: Currently, when authentication is turned on, the standalone cluster manager passes auth

[jira] [Created] (SPARK-8129) Securely pass auth secret to executors in standalone cluster mode

Kan Zhang created SPARK-8129: Summary: Securely pass auth secret to executors in standalone cluster mode Key: SPARK-8129 URL: https://issues.apache.org/jira/browse/SPARK-8129 Project: Spark

[jira] [Updated] (SPARK-8129) Securely pass auth secret to executors in standalone cluster mode

[ https://issues.apache.org/jira/browse/SPARK-8129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated SPARK-8129: - Description: Currently, when authentication is turned on, Worker passes auth secret to executors (also

[jira] [Updated] (SPARK-8129) Securely pass auth secret to executors in standalone cluster mode

[ https://issues.apache.org/jira/browse/SPARK-8129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated SPARK-8129: - Description: Currently, when authentication is turned on, cluster manager passes auth secrets to

[jira] [Updated] (SPARK-8129) Securely pass auth secret to executors in standalone cluster mode

[ https://issues.apache.org/jira/browse/SPARK-8129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated SPARK-8129: - Description: Currently, when authentication is turned on, Worker passes auth secret to executors (also

[jira] [Updated] (SPARK-8129) Securely pass auth secrets to executors in standalone cluster mode

[ https://issues.apache.org/jira/browse/SPARK-8129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated SPARK-8129: - Summary: Securely pass auth secrets to executors in standalone cluster mode (was: Securely pass auth

[jira] [Updated] (SPARK-1475) Drain event logging queue before stopping event logger

[ https://issues.apache.org/jira/browse/SPARK-1475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated SPARK-1475: - Summary: Drain event logging queue before stopping event logger (was: Draining event logging queue

[jira] [Updated] (SPARK-2736) Create Pyspark RDD from Apache Avro File

[ https://issues.apache.org/jira/browse/SPARK-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated SPARK-2736: - Summary: Create Pyspark RDD from Apache Avro File (was: Ceeate Pyspark RDD from Apache Avro File)

[jira] [Updated] (SPARK-2736) Create PySpark RDD from Apache Avro File

[ https://issues.apache.org/jira/browse/SPARK-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated SPARK-2736: - Summary: Create PySpark RDD from Apache Avro File (was: Create Pyspark RDD from Apache Avro File)

[jira] [Commented] (SPARK-2141) Add sc.getPersistentRDDs() to PySpark

[ https://issues.apache.org/jira/browse/SPARK-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14076262#comment-14076262 ] Kan Zhang commented on SPARK-2141: -- Hi [~nchammas], we are debating potential use cases

[jira] [Commented] (SPARK-1687) Support NamedTuples in RDDs

[ https://issues.apache.org/jira/browse/SPARK-1687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14077019#comment-14077019 ] Kan Zhang commented on SPARK-1687: -- Sure, pls go ahead and feel free to take over this

[jira] [Commented] (SPARK-1866) Closure cleaner does not null shadowed fields when outer scope is referenced

[ https://issues.apache.org/jira/browse/SPARK-1866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061501#comment-14061501 ] Kan Zhang commented on SPARK-1866: -- My previous comment may be less readable, let me try

[jira] [Commented] (SPARK-2024) Add saveAsSequenceFile to PySpark

[ https://issues.apache.org/jira/browse/SPARK-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14055912#comment-14055912 ] Kan Zhang commented on SPARK-2024: -- https://github.com/apache/spark/pull/1338 Add

[jira] [Commented] (SPARK-2010) Support for nested data in PySpark SQL

[ https://issues.apache.org/jira/browse/SPARK-2010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14052068#comment-14052068 ] Kan Zhang commented on SPARK-2010: -- Sounds reasonable. Named tuple is a better fit than

[jira] [Resolved] (SPARK-2013) Add Python pickleFile to programming guide

[ https://issues.apache.org/jira/browse/SPARK-2013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang resolved SPARK-2013. -- Resolution: Fixed Add Python pickleFile to programming guide

[jira] [Commented] (SPARK-2141) Add sc.getPersistentRDDs() to PySpark

[ https://issues.apache.org/jira/browse/SPARK-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14031356#comment-14031356 ] Kan Zhang commented on SPARK-2141: -- https://github.com/apache/spark/pull/1082 Add

[jira] [Updated] (SPARK-2079) Support batching when serializing SchemaRDD to Python

[ https://issues.apache.org/jira/browse/SPARK-2079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated SPARK-2079: - Summary: Support batching when serializing SchemaRDD to Python (was: Removing unnecessary wrapping when

[jira] [Commented] (SPARK-2010) Support for nested data in PySpark SQL

[ https://issues.apache.org/jira/browse/SPARK-2010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14027127#comment-14027127 ] Kan Zhang commented on SPARK-2010: -- PR: https://github.com/apache/spark/pull/1041

[jira] [Created] (SPARK-2079) Skip unnecessary wrapping in List when serializing SchemaRDD to Python

Kan Zhang created SPARK-2079: Summary: Skip unnecessary wrapping in List when serializing SchemaRDD to Python Key: SPARK-2079 URL: https://issues.apache.org/jira/browse/SPARK-2079 Project: Spark

[jira] [Commented] (SPARK-2079) Skip unnecessary wrapping in List when serializing SchemaRDD to Python

[ https://issues.apache.org/jira/browse/SPARK-2079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14025363#comment-14025363 ] Kan Zhang commented on SPARK-2079: -- PR: https://github.com/apache/spark/pull/1023 Skip

[jira] [Updated] (SPARK-2079) Removing unnecessary wrapping when serializing SchemaRDD to Python

[ https://issues.apache.org/jira/browse/SPARK-2079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated SPARK-2079: - Summary: Removing unnecessary wrapping when serializing SchemaRDD to Python (was: Skip unnecessary

[jira] [Issue Comment Deleted] (SPARK-2024) Add saveAsSequenceFile to PySpark

[ https://issues.apache.org/jira/browse/SPARK-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated SPARK-2024: - Comment: was deleted (was: You meant SPARK-1416?) Add saveAsSequenceFile to PySpark

[jira] [Issue Comment Deleted] (SPARK-937) Executors that exit cleanly should not have KILLED status

[ https://issues.apache.org/jira/browse/SPARK-937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated SPARK-937: Comment: was deleted (was: Hi Aaron, are you still working on this one? If not, could you assign it to me?

[jira] [Commented] (SPARK-937) Executors that exit cleanly should not have KILLED status

[ https://issues.apache.org/jira/browse/SPARK-937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14019309#comment-14019309 ] Kan Zhang commented on SPARK-937: - PR: https://github.com/apache/spark/pull/306 Executors

[jira] [Commented] (SPARK-1817) RDD zip erroneous when partitions do not divide RDD count

[ https://issues.apache.org/jira/browse/SPARK-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017812#comment-14017812 ] Kan Zhang commented on SPARK-1817: -- There are 2 issues related to this bug. One is that

[jira] [Issue Comment Deleted] (SPARK-1817) RDD zip erroneous when partitions do not divide RDD count

[ https://issues.apache.org/jira/browse/SPARK-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated SPARK-1817: - Comment: was deleted (was: PR: https://github.com/apache/spark/pull/760) RDD zip erroneous when

[jira] [Commented] (SPARK-2024) Add saveAsSequenceFile to PySpark

[ https://issues.apache.org/jira/browse/SPARK-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018485#comment-14018485 ] Kan Zhang commented on SPARK-2024: -- You meant SPARK-1416? Add saveAsSequenceFile to

[jira] [Updated] (SPARK-1519) support minPartitions parameter of wholeTextFiles() in pyspark

[ https://issues.apache.org/jira/browse/SPARK-1519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated SPARK-1519: - Fix Version/s: 1.0.1 1.1.0 support minPartitions parameter of wholeTextFiles() in

[jira] [Commented] (SPARK-937) Executors that exit cleanly should not have KILLED status

[ https://issues.apache.org/jira/browse/SPARK-937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13992631#comment-13992631 ] Kan Zhang commented on SPARK-937: - Hi Aaron, are you still working on this one? If not,

[jira] [Commented] (SPARK-1817) RDD zip erroneous when partitions do not divide RDD count

[ https://issues.apache.org/jira/browse/SPARK-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998153#comment-13998153 ] Kan Zhang commented on SPARK-1817: -- I opened SPARK-1837 as a specific fix for the error

[jira] [Commented] (SPARK-1161) Add saveAsObjectFile and SparkContext.objectFile in Python

[ https://issues.apache.org/jira/browse/SPARK-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996081#comment-13996081 ] Kan Zhang commented on SPARK-1161: -- PR: https://github.com/apache/spark/pull/755 Add

[jira] [Commented] (SPARK-1817) RDD zip erroneous when partitions do not divide RDD count

[ https://issues.apache.org/jira/browse/SPARK-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996818#comment-13996818 ] Kan Zhang commented on SPARK-1817: -- PR: https://github.com/apache/spark/pull/760 RDD

[jira] [Updated] (SPARK-1817) RDD zip erroneous when partitions do not divide RDD count

[ https://issues.apache.org/jira/browse/SPARK-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated SPARK-1817: - Affects Version/s: 1.0.0 RDD zip erroneous when partitions do not divide RDD count

[jira] [Commented] (SPARK-1519) support minPartitions parameter of wholeTextFiles() in pyspark

[ https://issues.apache.org/jira/browse/SPARK-1519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13992980#comment-13992980 ] Kan Zhang commented on SPARK-1519: -- PR: https://github.com/apache/spark/pull/697

[jira] [Assigned] (SPARK-1687) Support NamedTuples in RDDs

[ https://issues.apache.org/jira/browse/SPARK-1687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang reassigned SPARK-1687: Assignee: Kan Zhang Support NamedTuples in RDDs ---

[jira] [Updated] (SPARK-1604) Couldn't run spark-submit with yarn cluster mode when built with assemble-deps

[ https://issues.apache.org/jira/browse/SPARK-1604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated SPARK-1604: - Summary: Couldn't run spark-submit with yarn cluster mode when built with assemble-deps (was: Couldn't

[jira] [Updated] (SPARK-1604) Couldn't run spark-submit with yarn cluster mode when built with assemble-deps

[ https://issues.apache.org/jira/browse/SPARK-1604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated SPARK-1604: - Description: {code}

[jira] [Commented] (SPARK-1604) YARN cluster mode broken

[ https://issues.apache.org/jira/browse/SPARK-1604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13979624#comment-13979624 ] Kan Zhang commented on SPARK-1604: -- I doubt it, since when I ran it in YARN client mode,

[jira] [Updated] (SPARK-1604) YARN cluster mode broken

[ https://issues.apache.org/jira/browse/SPARK-1604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated SPARK-1604: - Component/s: (was: YARN) Build YARN cluster mode broken

[jira] [Comment Edited] (SPARK-1604) Couldn't run spark-submit with yarn cluster mode when using deps jar

[ https://issues.apache.org/jira/browse/SPARK-1604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13979994#comment-13979994 ] Kan Zhang edited comment on SPARK-1604 at 4/24/14 5:38 PM: --- Ah,

[jira] [Updated] (SPARK-1604) Couldn't run spark-submit with yarn cluster mode when using deps jar

[ https://issues.apache.org/jira/browse/SPARK-1604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated SPARK-1604: - Priority: Major (was: Blocker) Couldn't run spark-submit with yarn cluster mode when using deps jar

[jira] [Commented] (SPARK-1604) Couldn't run spark-submit with yarn cluster mode when using deps jar

[ https://issues.apache.org/jira/browse/SPARK-1604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13980003#comment-13980003 ] Kan Zhang commented on SPARK-1604: -- Sure, lowered it to Major. Couldn't run

[jira] [Commented] (SPARK-1571) UnresolvedException when running JavaSparkSQL example

[ https://issues.apache.org/jira/browse/SPARK-1571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13980034#comment-13980034 ] Kan Zhang commented on SPARK-1571: -- Thanks, it worked. UnresolvedException when running

[jira] [Commented] (SPARK-1570) Class loading issue when using Spark SQL Java API

[ https://issues.apache.org/jira/browse/SPARK-1570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977279#comment-13977279 ] Kan Zhang commented on SPARK-1570: -- PR: https://github.com/apache/spark/pull/484 Class

[jira] [Updated] (SPARK-1571) UnresolvedException when running JavaSparkSQL example

[ https://issues.apache.org/jira/browse/SPARK-1571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated SPARK-1571: - Description: When running JavaSparkSQL example using spark-submit in local mode (this happens after

[jira] [Updated] (SPARK-1570) Class loading issue when using Spark SQL Java API

[ https://issues.apache.org/jira/browse/SPARK-1570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated SPARK-1570: - Description: ClassNotFoundException in Executor when running JavaSparkSQL example using spark-submit in

[jira] [Created] (SPARK-1571) UnresolvedException when running JavaSparkSQL example

Kan Zhang created SPARK-1571: Summary: UnresolvedException when running JavaSparkSQL example Key: SPARK-1571 URL: https://issues.apache.org/jira/browse/SPARK-1571 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-1475) Draining event logging queue before stopping event logger

[ https://issues.apache.org/jira/browse/SPARK-1475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13972225#comment-13972225 ] Kan Zhang commented on SPARK-1475: -- A second PR that fixes the unit test introduced

[jira] [Updated] (SPARK-1475) Draining event logging queue before stopping event logger

[ https://issues.apache.org/jira/browse/SPARK-1475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated SPARK-1475: - Affects Version/s: 1.0.0 Draining event logging queue before stopping event logger

[jira] [Resolved] (SPARK-1475) Draining event logging queue before stopping event logger

[ https://issues.apache.org/jira/browse/SPARK-1475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang resolved SPARK-1475. -- Resolution: Fixed Draining event logging queue before stopping event logger

[jira] [Updated] (SPARK-1475) Draining event logging queue before stopping event logger

[ https://issues.apache.org/jira/browse/SPARK-1475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated SPARK-1475: - Description: When stopping SparkListenerBus, its event queue needs to be drained. And this needs to

[jira] [Assigned] (SPARK-1460) Set operations on SchemaRDDs are needlessly destructive of schema information.

[ https://issues.apache.org/jira/browse/SPARK-1460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang reassigned SPARK-1460: Assignee: Kan Zhang Set operations on SchemaRDDs are needlessly destructive of schema

[jira] [Commented] (SPARK-1348) Spark UI's do not bind to localhost interface anymore

[ https://issues.apache.org/jira/browse/SPARK-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959466#comment-13959466 ] Kan Zhang commented on SPARK-1348: -- JettyUtils.startJettyServer() used to bind to all

[jira] [Commented] (SPARK-1118) Executor state shows as KILLED even the application is finished normally

[ https://issues.apache.org/jira/browse/SPARK-1118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13958329#comment-13958329 ] Kan Zhang commented on SPARK-1118: -- I took a look running SparkPi on my single node