[jira] [Commented] (SPARK-21560) Add hold mode for the LiveListenerBus

2017-07-28 Thread Li Yuanjian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106030#comment-16106030 ] Li Yuanjian commented on SPARK-21560: - Something wrong with the sync between PR and JIRA, the PR

[jira] [Commented] (SPARK-21573) Tests failing with run-tests.py SyntaxError occasionally in Jenkins

2017-07-28 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106029#comment-16106029 ] Dongjoon Hyun commented on SPARK-21573: --- Thank you for pinging me, [~hyukjin.kwon]. +1 for the

[jira] [Commented] (SPARK-21573) Tests failing with run-tests.py SyntaxError occasionally in Jenkins

2017-07-28 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106024#comment-16106024 ] holdenk commented on SPARK-21573: - Yes we did drop 2.6 support. We should change the script to python2.7

[jira] [Updated] (SPARK-21573) Tests failing with run-tests.py SyntaxError occasionally in Jenkins

2017-07-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-21573: - Description: It looks default {{python}} in the path at few places such as {{./dev/run-tests}}

[jira] [Updated] (SPARK-21573) Tests failing with run-tests.py SyntaxError occasionally in Jenkins

2017-07-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-21573: - Description: {code} python2.6 run-tests.py File "run-tests.py", line 124 {m:

[jira] [Commented] (SPARK-21573) Tests failing with run-tests.py SyntaxError occasionally in Jenkins

2017-07-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106021#comment-16106021 ] Hyukjin Kwon commented on SPARK-21573: -- I think we officially dropped Python 2.6 in

[jira] [Updated] (SPARK-21573) Tests failing with run-tests.py SyntaxError occasionally in Jenkins

2017-07-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-21573: - Summary: Tests failing with run-tests.py SyntaxError occasionally in Jenkins (was: Tests

[jira] [Updated] (SPARK-21573) Tests failing with run-tests.py SyntaxError occasionally

2017-07-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-21573: - Summary: Tests failing with run-tests.py SyntaxError occasionally (was: run-tests script fails

[jira] [Assigned] (SPARK-20090) Add StructType.fieldNames to Python API

2017-07-28 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-20090: --- Assignee: Hyukjin Kwon > Add StructType.fieldNames to Python API >

[jira] [Created] (SPARK-21573) run-tests script fails with Python 2.6

2017-07-28 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-21573: Summary: run-tests script fails with Python 2.6 Key: SPARK-21573 URL: https://issues.apache.org/jira/browse/SPARK-21573 Project: Spark Issue Type: Test

[jira] [Resolved] (SPARK-20090) Add StructType.fieldNames to Python API

2017-07-28 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-20090. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18618

[jira] [Commented] (SPARK-21533) "configure(...)" method not called when using Hive Generic UDFs

2017-07-28 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106000#comment-16106000 ] Takeshi Yamamuro commented on SPARK-21533: -- I feel supporting this function makes little sense

[jira] [Created] (SPARK-21572) Add description on how to exit the spark-shell in the welcome message

2017-07-28 Thread Donghui Xu (JIRA)
Donghui Xu created SPARK-21572: -- Summary: Add description on how to exit the spark-shell in the welcome message Key: SPARK-21572 URL: https://issues.apache.org/jira/browse/SPARK-21572 Project: Spark

[jira] [Commented] (SPARK-19490) Hive partition columns are case-sensitive

2017-07-28 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105979#comment-16105979 ] cen yuhai commented on SPARK-19490: --- I forgot the `fixed PR`, maybe that pr will not by backport to 2.1

[jira] [Created] (SPARK-21571) Spark history server leaves incomplete or unreadable history files around forever.

2017-07-28 Thread Eric Vandenberg (JIRA)
Eric Vandenberg created SPARK-21571: --- Summary: Spark history server leaves incomplete or unreadable history files around forever. Key: SPARK-21571 URL: https://issues.apache.org/jira/browse/SPARK-21571

[jira] [Commented] (SPARK-18535) Redact sensitive information from Spark logs and UI

2017-07-28 Thread Diogo Munaro Vieira (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105955#comment-16105955 ] Diogo Munaro Vieira commented on SPARK-18535: - I did a merge request for version 2.1.2:

[jira] [Commented] (SPARK-19720) Redact sensitive information from SparkSubmit console output

2017-07-28 Thread Diogo Munaro Vieira (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105952#comment-16105952 ] Diogo Munaro Vieira commented on SPARK-19720: - I did a merge request for this compatibility

[jira] [Commented] (SPARK-20433) Security issue with jackson-databind

2017-07-28 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105951#comment-16105951 ] Andrew Ash commented on SPARK-20433: As I wrote in that PR, it's 2.6.7.1 of jackson-databind that has

[jira] [Commented] (SPARK-21274) Implement EXCEPT ALL and INTERSECT ALL

2017-07-28 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105919#comment-16105919 ] Liang-Chi Hsieh commented on SPARK-21274: - [~Tagar] I've tried the query on PostgreSQL, the

[jira] [Commented] (SPARK-21570) File __spark_libs__XXX.zip does not exist on networked file system w/ yarn

2017-07-28 Thread Albert Chu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105900#comment-16105900 ] Albert Chu commented on SPARK-21570: Oh, and because it will likely be asked and may likely be

[jira] [Created] (SPARK-21570) File __spark_libs__XXX.zip does not exist on networked file system w/ yarn

2017-07-28 Thread Albert Chu (JIRA)
Albert Chu created SPARK-21570: -- Summary: File __spark_libs__XXX.zip does not exist on networked file system w/ yarn Key: SPARK-21570 URL: https://issues.apache.org/jira/browse/SPARK-21570 Project:

[jira] [Created] (SPARK-21569) Internal Spark class needs to be kryo-registered

2017-07-28 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-21569: - Summary: Internal Spark class needs to be kryo-registered Key: SPARK-21569 URL: https://issues.apache.org/jira/browse/SPARK-21569 Project: Spark Issue

[jira] [Commented] (SPARK-20433) Security issue with jackson-databind

2017-07-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105792#comment-16105792 ] Sean Owen commented on SPARK-20433: --- You updated to 2.6.7 but indicated above that's still vulnerable.

[jira] [Comment Edited] (SPARK-21549) Spark fails to complete job correctly in case of OutputFormat which do not write into hdfs

2017-07-28 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105630#comment-16105630 ] Mridul Muralidharan edited comment on SPARK-21549 at 7/28/17 10:16 PM:

[jira] [Comment Edited] (SPARK-21549) Spark fails to complete job correctly in case of OutputFormat which do not write into hdfs

2017-07-28 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105630#comment-16105630 ] Mridul Muralidharan edited comment on SPARK-21549 at 7/28/17 10:14 PM:

[jira] [Commented] (SPARK-21563) Race condition when serializing TaskDescriptions and adding jars

2017-07-28 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105772#comment-16105772 ] Andrew Ash commented on SPARK-21563: And for reference, I added this additional logging to assist in

[jira] [Commented] (SPARK-20433) Security issue with jackson-databind

2017-07-28 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105751#comment-16105751 ] Andrew Ash commented on SPARK-20433: Here's the patch I put in my fork of Spark:

[jira] [Created] (SPARK-21568) ConsoleProgressBar should only be enabled in shells

2017-07-28 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-21568: -- Summary: ConsoleProgressBar should only be enabled in shells Key: SPARK-21568 URL: https://issues.apache.org/jira/browse/SPARK-21568 Project: Spark

[jira] [Commented] (SPARK-19490) Hive partition columns are case-sensitive

2017-07-28 Thread Taklon Stephen Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105734#comment-16105734 ] Taklon Stephen Wu commented on SPARK-19490: --- https://github.com/apache/spark/pull/16832 is

[jira] [Created] (SPARK-21567) Dataset with Tuple of type alias throws error

2017-07-28 Thread Tomasz Bartczak (JIRA)
Tomasz Bartczak created SPARK-21567: --- Summary: Dataset with Tuple of type alias throws error Key: SPARK-21567 URL: https://issues.apache.org/jira/browse/SPARK-21567 Project: Spark Issue

[jira] [Created] (SPARK-21566) Python method for summary

2017-07-28 Thread Andrew Ray (JIRA)
Andrew Ray created SPARK-21566: -- Summary: Python method for summary Key: SPARK-21566 URL: https://issues.apache.org/jira/browse/SPARK-21566 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-21549) Spark fails to complete job correctly in case of OutputFormat which do not write into hdfs

2017-07-28 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105630#comment-16105630 ] Mridul Muralidharan commented on SPARK-21549: - This affects both mapred

[jira] [Updated] (SPARK-21565) aggregate query fails with watermark on eventTime but works with watermark on timestamp column generated by current_timestamp

2017-07-28 Thread Amit Assudani (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Assudani updated SPARK-21565: -- Description: *Short Description: * Aggregation query fails with eventTime as watermark column

[jira] [Created] (SPARK-21565) aggregate query fails with watermark on eventTime but works with watermark on timestamp column generated by current_timestamp

2017-07-28 Thread Amit Assudani (JIRA)
Amit Assudani created SPARK-21565: - Summary: aggregate query fails with watermark on eventTime but works with watermark on timestamp column generated by current_timestamp Key: SPARK-21565 URL:

[jira] [Updated] (SPARK-21564) TaskDescription decoding failure should fail the task

2017-07-28 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-21564: --- Description: cc [~robert3005] I was seeing an issue where Spark was throwing this exception:

[jira] [Updated] (SPARK-21563) Race condition when serializing TaskDescriptions and adding jars

2017-07-28 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-21563: --- Description: cc [~robert3005] I was seeing this exception during some running Spark jobs:

[jira] [Created] (SPARK-21564) TaskDescription decoding failure should fail the task

2017-07-28 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-21564: -- Summary: TaskDescription decoding failure should fail the task Key: SPARK-21564 URL: https://issues.apache.org/jira/browse/SPARK-21564 Project: Spark Issue

[jira] [Created] (SPARK-21563) Race condition when serializing TaskDescriptions and adding jars

2017-07-28 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-21563: -- Summary: Race condition when serializing TaskDescriptions and adding jars Key: SPARK-21563 URL: https://issues.apache.org/jira/browse/SPARK-21563 Project: Spark

[jira] [Commented] (SPARK-18016) Code Generation: Constant Pool Past Limit for Wide/Nested Dataset

2017-07-28 Thread James Conner (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105528#comment-16105528 ] James Conner commented on SPARK-18016: -- Thank you for letting me know, Kazuaki! Please let me know

[jira] [Commented] (SPARK-19720) Redact sensitive information from SparkSubmit console output

2017-07-28 Thread Diogo Munaro Vieira (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105514#comment-16105514 ] Diogo Munaro Vieira commented on SPARK-19720: - Yes, but it's a major security bug as

[jira] [Commented] (SPARK-21562) Spark may request extra containers if the rpc between YARN and spark is too fast

2017-07-28 Thread Wei Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105505#comment-16105505 ] Wei Chen commented on SPARK-21562: -- Thanks for comments, I will close this one. On Fri, Jul 28, 2017 at

[jira] [Commented] (SPARK-19938) java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field

2017-07-28 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105449#comment-16105449 ] Ryan Blue commented on SPARK-19938: --- [~snavatski], I just hit this problem also and found out what

[jira] [Commented] (SPARK-17614) sparkSession.read() .jdbc(***) use the sql syntax "where 1=0" that Cassandra does not support

2017-07-28 Thread Paul Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105439#comment-16105439 ] Paul Wu commented on SPARK-17614: - Oh, sorry. I thought I could use a query hereas I do with other

[jira] [Commented] (SPARK-17614) sparkSession.read() .jdbc(***) use the sql syntax "where 1=0" that Cassandra does not support

2017-07-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105431#comment-16105431 ] Sean Owen commented on SPARK-17614: --- Well, it's unrelated to this issue, so this isn't the place. And

[jira] [Commented] (SPARK-19720) Redact sensitive information from SparkSubmit console output

2017-07-28 Thread Mark Grover (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105427#comment-16105427 ] Mark Grover commented on SPARK-19720: - I wasn't planning on. One could argue the case that this could

[jira] [Commented] (SPARK-21546) dropDuplicates with watermark yields RuntimeException due to binding failure

2017-07-28 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105426#comment-16105426 ] Shixiong Zhu commented on SPARK-21546: -- Yeah, good catch. The watermark column should be one of the

[jira] [Commented] (SPARK-17614) sparkSession.read() .jdbc(***) use the sql syntax "where 1=0" that Cassandra does not support

2017-07-28 Thread Paul Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105425#comment-16105425 ] Paul Wu commented on SPARK-17614: - So create a new issue? Or this is not an issue to you? >

[jira] [Updated] (SPARK-21523) Fix bug of strong wolfe linesearch `init` parameter lose effectiveness

2017-07-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21523: -- Target Version/s: 2.2.1 Priority: Critical (was: Minor) I think this is fairly critical

[jira] [Comment Edited] (SPARK-17614) sparkSession.read() .jdbc(***) use the sql syntax "where 1=0" that Cassandra does not support

2017-07-28 Thread Paul Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105402#comment-16105402 ] Paul Wu edited comment on SPARK-17614 at 7/28/17 6:14 PM: -- The fix does not

[jira] [Resolved] (SPARK-21561) spark-streaming-kafka-010 DSteam is not pulling anything from Kafka

2017-07-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21561. --- Resolution: Invalid This isn't a place to ask for input on your code -- you'd have to show a

[jira] [Resolved] (SPARK-17614) sparkSession.read() .jdbc(***) use the sql syntax "where 1=0" that Cassandra does not support

2017-07-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-17614. --- Resolution: Fixed [~zwu@gmail.com] please don't reopen this > sparkSession.read() .jdbc(***)

[jira] [Closed] (SPARK-17614) sparkSession.read() .jdbc(***) use the sql syntax "where 1=0" that Cassandra does not support

2017-07-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen closed SPARK-17614. - > sparkSession.read() .jdbc(***) use the sql syntax "where 1=0" that Cassandra > does not support >

[jira] [Comment Edited] (SPARK-17614) sparkSession.read() .jdbc(***) use the sql syntax "where 1=0" that Cassandra does not support

2017-07-28 Thread Paul Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105402#comment-16105402 ] Paul Wu edited comment on SPARK-17614 at 7/28/17 6:09 PM: -- The fix does not

[jira] [Reopened] (SPARK-17614) sparkSession.read() .jdbc(***) use the sql syntax "where 1=0" that Cassandra does not support

2017-07-28 Thread Paul Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Wu reopened SPARK-17614: - The fix does not support the syntax on the syntax like this: {{.jdbc(JDBC_URL, "(select * from emp)",

[jira] [Comment Edited] (SPARK-17614) sparkSession.read() .jdbc(***) use the sql syntax "where 1=0" that Cassandra does not support

2017-07-28 Thread Paul Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105402#comment-16105402 ] Paul Wu edited comment on SPARK-17614 at 7/28/17 6:08 PM: -- The fix does not

[jira] [Resolved] (SPARK-21562) Spark may request extra containers if the rpc between YARN and spark is too fast

2017-07-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-21562. Resolution: Duplicate > Spark may request extra containers if the rpc between YARN and

[jira] [Updated] (SPARK-21562) Spark may request extra containers if the rpc between YARN and spark is too fast

2017-07-28 Thread Wei Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Chen updated SPARK-21562: - Description: hi huys, I find an interesting problem when spark tries to request containers from YARN.

[jira] [Updated] (SPARK-21562) Spark may request extra containers if the rpc between YARN and spark is too fast

2017-07-28 Thread Wei Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Chen updated SPARK-21562: - Description: hi huys, I find an interesting problem when spark tries to request containers from YARN.

[jira] [Updated] (SPARK-21562) Spark may request extra containers if the rpc between YARN and spark is too fast

2017-07-28 Thread Wei Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Chen updated SPARK-21562: - Description: hi huys, I find an interesting problem when spark tries to request containers from YARN.

[jira] [Updated] (SPARK-21562) Spark may request extra containers if the rpc between YARN and spark is too fast

2017-07-28 Thread Wei Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Chen updated SPARK-21562: - Description: hi huys, I find an interesting problem when spark tries to request containers from YARN.

[jira] [Updated] (SPARK-21562) Spark may request extra containers if the rpc between YARN and spark is too fast

2017-07-28 Thread Wei Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Chen updated SPARK-21562: - Description: hi huys, I find an interesting problem when spark tries to request containers from YARN.

[jira] [Updated] (SPARK-21562) Spark may request extra containers if the rpc between YARN and spark is too fast

2017-07-28 Thread Wei Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Chen updated SPARK-21562: - Description: hi huys, I find an interesting problem when spark tries to request containers from YARN.

[jira] [Updated] (SPARK-21562) Spark may request extra containers if the rpc between YARN and spark is too fast

2017-07-28 Thread Wei Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Chen updated SPARK-21562: - Description: hi huys, I find an interesting problem when spark tries to request containers from YARN.

[jira] [Updated] (SPARK-21562) Spark may request extra containers if the rpc between YARN and spark is too fast

2017-07-28 Thread Wei Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Chen updated SPARK-21562: - Description: hi huys, I find an interesting problem when spark tries to request containers from YARN.

[jira] [Updated] (SPARK-21562) Spark may request extra containers if the rpc between YARN and spark is too fast

2017-07-28 Thread Wei Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Chen updated SPARK-21562: - Description: hi huys, I find an interesting problem when spark tries to request containers from YARN.

[jira] [Created] (SPARK-21562) Spark may request extra containers if the rpc between YARN and spark is too fast

2017-07-28 Thread Wei Chen (JIRA)
Wei Chen created SPARK-21562: Summary: Spark may request extra containers if the rpc between YARN and spark is too fast Key: SPARK-21562 URL: https://issues.apache.org/jira/browse/SPARK-21562 Project:

[jira] [Created] (SPARK-21561) spark-streaming-kafka-010 DSteam is not pulling anything from Kafka

2017-07-28 Thread Vlad Badelita (JIRA)
Vlad Badelita created SPARK-21561: - Summary: spark-streaming-kafka-010 DSteam is not pulling anything from Kafka Key: SPARK-21561 URL: https://issues.apache.org/jira/browse/SPARK-21561 Project: Spark

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2017-07-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105164#comment-16105164 ] Thomas Graves commented on SPARK-17321: --- Can you clarify? as stated above you should not be using

[jira] [Comment Edited] (SPARK-21274) Implement EXCEPT ALL and INTERSECT ALL

2017-07-28 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105127#comment-16105127 ] Ruslan Dautkhanov edited comment on SPARK-21274 at 7/28/17 3:47 PM:

[jira] [Comment Edited] (SPARK-21274) Implement EXCEPT ALL and INTERSECT ALL

2017-07-28 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105127#comment-16105127 ] Ruslan Dautkhanov edited comment on SPARK-21274 at 7/28/17 3:46 PM:

[jira] [Commented] (SPARK-21541) Spark Logs show incorrect job status for a job that does not create SparkContext

2017-07-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105133#comment-16105133 ] Thomas Graves commented on SPARK-21541: --- it was merged

[jira] [Commented] (SPARK-21274) Implement EXCEPT ALL and INTERSECT ALL

2017-07-28 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105127#comment-16105127 ] Ruslan Dautkhanov commented on SPARK-21274: --- [~viirya], yes it returns {noformat}[1, 2,

[jira] [Commented] (SPARK-21541) Spark Logs show incorrect job status for a job that does not create SparkContext

2017-07-28 Thread Parth Gandhi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105116#comment-16105116 ] Parth Gandhi commented on SPARK-21541: -- The change has been merged. Thank you. > Spark Logs show

[jira] [Updated] (SPARK-21549) Spark fails to complete job correctly in case of OutputFormat which do not write into hdfs

2017-07-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21549: -- Priority: Major (was: Blocker) > Spark fails to complete job correctly in case of OutputFormat which

[jira] [Updated] (SPARK-21549) Spark fails to complete job correctly in case of OutputFormat which do not write into hdfs

2017-07-28 Thread Sergey Zhemzhitsky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Zhemzhitsky updated SPARK-21549: --- Priority: Blocker (was: Critical) > Spark fails to complete job correctly in case

[jira] [Commented] (SPARK-21190) SPIP: Vectorized UDFs in Python

2017-07-28 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105095#comment-16105095 ] Li Jin commented on SPARK-21190: I think the use case 2 of what [~rxin] proposed originally is a good API

[jira] [Commented] (SPARK-21555) GROUP BY don't work with expressions with NVL and nested objects

2017-07-28 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105092#comment-16105092 ] Liang-Chi Hsieh commented on SPARK-21555: - The sync between PR and JIRA seems broken still. I

[jira] [Commented] (SPARK-21190) SPIP: Vectorized UDFs in Python

2017-07-28 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105072#comment-16105072 ] Li Jin commented on SPARK-21190: [~cloud_fan], thanks for pointing out `ArrowColumnVector`. [~bryanc], I

[jira] [Comment Edited] (SPARK-21190) SPIP: Vectorized UDFs in Python

2017-07-28 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105072#comment-16105072 ] Li Jin edited comment on SPARK-21190 at 7/28/17 3:06 PM: - [~cloud_fan], thanks

[jira] [Commented] (SPARK-21541) Spark Logs show incorrect job status for a job that does not create SparkContext

2017-07-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105049#comment-16105049 ] Sean Owen commented on SPARK-21541: --- Was this change merged? I don't think it was

[jira] [Resolved] (SPARK-21541) Spark Logs show incorrect job status for a job that does not create SparkContext

2017-07-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21541. --- Resolution: Fixed Assignee: Parth Gandhi Fix Version/s: 2.3.0 > Spark Logs

[jira] [Assigned] (SPARK-21553) Add the description of the default value of master parameter in the spark-shell

2017-07-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-21553: - Assignee: Donghui Xu > Add the description of the default value of master parameter in the >

[jira] [Updated] (SPARK-21553) Add the description of the default value of master parameter in the spark-shell

2017-07-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21553: -- Priority: Trivial (was: Minor) > Add the description of the default value of master parameter in the

[jira] [Resolved] (SPARK-21553) Add the description of the default value of master parameter in the spark-shell

2017-07-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21553. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18755

[jira] [Issue Comment Deleted] (SPARK-14927) DataFrame. saveAsTable creates RDD partitions but not Hive partitions

2017-07-28 Thread SOMASUNDARAM SUDALAIMUTHU (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SOMASUNDARAM SUDALAIMUTHU updated SPARK-14927: -- Comment: was deleted (was: Is this fixed in 2.0 version ?) >

[jira] [Commented] (SPARK-14927) DataFrame. saveAsTable creates RDD partitions but not Hive partitions

2017-07-28 Thread SOMASUNDARAM SUDALAIMUTHU (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105004#comment-16105004 ] SOMASUNDARAM SUDALAIMUTHU commented on SPARK-14927: --- Is this fixed in 2.0 version ? >

[jira] [Created] (SPARK-21560) Add hold mode for the LiveListenerBus

2017-07-28 Thread Li Yuanjian (JIRA)
Li Yuanjian created SPARK-21560: --- Summary: Add hold mode for the LiveListenerBus Key: SPARK-21560 URL: https://issues.apache.org/jira/browse/SPARK-21560 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-21067) Thrift Server - CTAS fail with Unable to move source

2017-07-28 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104918#comment-16104918 ] Wenchen Fan commented on SPARK-21067: - We have many tests for CREATE TABLE inside Spark SQL, so I

[jira] [Updated] (SPARK-21306) OneVsRest Conceals Columns That May Be Relevant To Underlying Classifier

2017-07-28 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang updated SPARK-21306: Fix Version/s: (was: 2.1.2) (was: 2.0.3) > OneVsRest Conceals Columns

[jira] [Updated] (SPARK-20919) Simplificaiton of CachedKafkaConsumer.

2017-07-28 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-20919: Description: Using an object pool instead of a cache for recycling objects in Kafka

[jira] [Updated] (SPARK-20919) Simplificaiton of CachedKafkaConsumer.

2017-07-28 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-20919: Summary: Simplificaiton of CachedKafkaConsumer. (was: Simplificaiton of

[jira] [Updated] (SPARK-21559) Remove Mesos fine-grained mode

2017-07-28 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stavros Kontopoulos updated SPARK-21559: Description: After discussing this with people from Mesosphere we agreed that it

[jira] [Updated] (SPARK-21559) Remove Mesos fine-grained mode

2017-07-28 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stavros Kontopoulos updated SPARK-21559: Summary: Remove Mesos fine-grained mode (was: Remove Mesos Fine-grain mode) >

[jira] [Created] (SPARK-21559) Remove Mesos Fine-grain mode

2017-07-28 Thread Stavros Kontopoulos (JIRA)
Stavros Kontopoulos created SPARK-21559: --- Summary: Remove Mesos Fine-grain mode Key: SPARK-21559 URL: https://issues.apache.org/jira/browse/SPARK-21559 Project: Spark Issue Type: Task

[jira] [Updated] (SPARK-21553) Add the description of the default value of master parameter in the spark-shell

2017-07-28 Thread Donghui Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Donghui Xu updated SPARK-21553: --- Summary: Add the description of the default value of master parameter in the spark-shell (was:

[jira] [Commented] (SPARK-21479) Outer join filter pushdown in null supplying table when condition is on one of the joined columns

2017-07-28 Thread Abhijit Bhole (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104708#comment-16104708 ] Abhijit Bhole commented on SPARK-21479: --- So here is the actual use case - {code:java} spark =

[jira] [Created] (SPARK-21558) Kinesis lease failover time should be increased or made configurable

2017-07-28 Thread JIRA
Clément MATHIEU created SPARK-21558: --- Summary: Kinesis lease failover time should be increased or made configurable Key: SPARK-21558 URL: https://issues.apache.org/jira/browse/SPARK-21558 Project:

[jira] [Commented] (SPARK-21554) Spark Hive reporting pyspark.sql.utils.AnalysisException: u'Table not found: XXX' when run on yarn cluster

2017-07-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104640#comment-16104640 ] Sean Owen commented on SPARK-21554: --- The error here doesn't show the actual error. > Spark Hive

[jira] [Resolved] (SPARK-21557) Debug issues for SparkML(scala.Predef$.any2ArrowAssoc)

2017-07-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21557. --- Resolution: Invalid Fix Version/s: (was: 2.1.2) Target Version/s: (was: 2.2.0)

[jira] [Commented] (SPARK-21554) Spark Hive reporting pyspark.sql.utils.AnalysisException: u'Table not found: XXX' when run on yarn cluster

2017-07-28 Thread Subhod Lagade (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104634#comment-16104634 ] Subhod Lagade commented on SPARK-21554: --- Thanks for quick reply @Hyukjin Kwon - We have spark

  1   2   >