[jira] [Assigned] (SPARK-20368) Support Sentry on PySpark workers

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20368: Assignee: Apache Spark > Support Sentry on PySpark workers >

[jira] [Created] (SPARK-20369) pyspark: Dynamic configuration with SparkConf does not work

2017-04-18 Thread Matthew McClain (JIRA)
Matthew McClain created SPARK-20369: --- Summary: pyspark: Dynamic configuration with SparkConf does not work Key: SPARK-20369 URL: https://issues.apache.org/jira/browse/SPARK-20369 Project: Spark

[jira] [Created] (SPARK-20368) Support Sentry on PySpark workers

2017-04-18 Thread Alexander Shorin (JIRA)
Alexander Shorin created SPARK-20368: Summary: Support Sentry on PySpark workers Key: SPARK-20368 URL: https://issues.apache.org/jira/browse/SPARK-20368 Project: Spark Issue Type: New

[jira] [Commented] (SPARK-20367) Spark silently escapes partition column names

2017-04-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972871#comment-15972871 ] Hyukjin Kwon commented on SPARK-20367: -- Doh. I rushed reading ... > Spark silently escapes

[jira] [Commented] (SPARK-20367) Spark silently escapes partition column names

2017-04-18 Thread Juliusz Sompolski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972867#comment-15972867 ] Juliusz Sompolski commented on SPARK-20367: --- Hi [~hyukjin.kwon]. I tested also with parquet,

[jira] [Commented] (SPARK-20367) Spark silently escapes partition column names

2017-04-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972865#comment-15972865 ] Hyukjin Kwon commented on SPARK-20367: -- Actually, I did while trying to reproduce this :) {code}

[jira] [Commented] (SPARK-20367) Spark silently escapes partition column names

2017-04-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972850#comment-15972850 ] Hyukjin Kwon commented on SPARK-20367: -- I guess probably this is not a CSV datasource specific

[jira] [Commented] (SPARK-20343) SBT master build for Hadoop 2.6 in Jenkins fails due to Avro version resolution

2017-04-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972842#comment-15972842 ] Hyukjin Kwon commented on SPARK-20343: -- Please let me know if anyone is able to reproduce this. I am

[jira] [Commented] (SPARK-6509) MDLP discretizer

2017-04-18 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972806#comment-15972806 ] Sergio Ramírez commented on SPARK-6509: --- Thanks again Barry for your support. I hope this proof can

[jira] [Created] (SPARK-20367) Spark silently escapes partition column names

2017-04-18 Thread Juliusz Sompolski (JIRA)
Juliusz Sompolski created SPARK-20367: - Summary: Spark silently escapes partition column names Key: SPARK-20367 URL: https://issues.apache.org/jira/browse/SPARK-20367 Project: Spark

[jira] [Assigned] (SPARK-20281) Table-valued function range in SQL should use the same number of partitions as spark.range

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20281: Assignee: Apache Spark > Table-valued function range in SQL should use the same number of

[jira] [Assigned] (SPARK-20281) Table-valued function range in SQL should use the same number of partitions as spark.range

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20281: Assignee: (was: Apache Spark) > Table-valued function range in SQL should use the

[jira] [Commented] (SPARK-20281) Table-valued function range in SQL should use the same number of partitions as spark.range

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972758#comment-15972758 ] Apache Spark commented on SPARK-20281: -- User 'maropu' has created a pull request for this issue:

[jira] [Commented] (SPARK-20356) Spark sql group by returns incorrect results after join + distinct transformations

2017-04-18 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972753#comment-15972753 ] Herman van Hovell commented on SPARK-20356: --- Here is a reproduction in scala: {noformat} val

[jira] [Commented] (SPARK-20281) Table-valued function range in SQL should use the same number of partitions as spark.range

2017-04-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972756#comment-15972756 ] Takeshi Yamamuro commented on SPARK-20281: -- IIUC they internally use the same value (that is,

[jira] [Commented] (SPARK-6509) MDLP discretizer

2017-04-18 Thread Barry Becker (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972744#comment-15972744 ] Barry Becker commented on SPARK-6509: - As further proof of relevance, I will be giving a

[jira] [Assigned] (SPARK-20366) Fix recursive join reordering: inside joins are not reordered

2017-04-18 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-20366: --- Assignee: Zhenhua Wang > Fix recursive join reordering: inside joins are not reordered >

[jira] [Resolved] (SPARK-20366) Fix recursive join reordering: inside joins are not reordered

2017-04-18 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-20366. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17668

[jira] [Commented] (SPARK-20356) Spark sql group by returns incorrect results after join + distinct transformations

2017-04-18 Thread Ed Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972544#comment-15972544 ] Ed Lee commented on SPARK-20356: really quite dangerous bug > Spark sql group by returns incorrect

[jira] [Commented] (SPARK-20343) SBT master build for Hadoop 2.6 in Jenkins fails due to Avro version resolution

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972490#comment-15972490 ] Apache Spark commented on SPARK-20343: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Commented] (SPARK-1548) Add Partial Random Forest algorithm to MLlib

2017-04-18 Thread Mohamed Baddar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972474#comment-15972474 ] Mohamed Baddar commented on SPARK-1548: --- [~srowen] [~josephkb] any updates on the possibility of

[jira] [Assigned] (SPARK-20344) Duplicate call in FairSchedulableBuilder.addTaskSetManager

2017-04-18 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-20344: - Assignee: Robert Stupp > Duplicate call in FairSchedulableBuilder.addTaskSetManager >

[jira] [Resolved] (SPARK-20344) Duplicate call in FairSchedulableBuilder.addTaskSetManager

2017-04-18 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-20344. --- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17647

[jira] [Resolved] (SPARK-20361) JVM locale affects SQL type names

2017-04-18 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-20361. --- Resolution: Duplicate > JVM locale affects SQL type names > -- > >

[jira] [Reopened] (SPARK-20361) JVM locale affects SQL type names

2017-04-18 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reopened SPARK-20361: --- > JVM locale affects SQL type names > -- > > Key:

[jira] [Commented] (SPARK-20364) Parquet predicate pushdown on columns with dots return empty results

2017-04-18 Thread Robert Kruszewski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972400#comment-15972400 ] Robert Kruszewski commented on SPARK-20364: --- Looks like parquet doesn't differentiate between

[jira] [Commented] (SPARK-20169) Groupby Bug with Sparksql

2017-04-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972401#comment-15972401 ] Takeshi Yamamuro commented on SPARK-20169: -- I also could reproduce this on /bin/pyspark in v2.1

[jira] [Comment Edited] (SPARK-20364) Parquet predicate pushdown on columns with dots return empty results

2017-04-18 Thread Robert Kruszewski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972400#comment-15972400 ] Robert Kruszewski edited comment on SPARK-20364 at 4/18/17 9:36 AM:

[jira] [Resolved] (SPARK-20363) sessionstate.get is get the same object in hive project, when I use spark-beeline

2017-04-18 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-20363. --- Resolution: Invalid This is very unclear. Add a comment if you can significantly clarify what this

[jira] [Commented] (SPARK-19995) Using real user to connect HiveMetastore in HiveClientImpl

2017-04-18 Thread meiyoula (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972373#comment-15972373 ] meiyoula commented on SPARK-19995: -- Will the token be expired? > Using real user to connect

[jira] [Comment Edited] (SPARK-20174) Analyzer gives mysterious AnalysisException when posexplode used in withColumn

2017-04-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972359#comment-15972359 ] Takeshi Yamamuro edited comment on SPARK-20174 at 4/18/17 9:09 AM: --- You

[jira] [Commented] (SPARK-20174) Analyzer gives mysterious AnalysisException when posexplode used in withColumn

2017-04-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972359#comment-15972359 ] Takeshi Yamamuro commented on SPARK-20174: -- You could fix this like

[jira] [Commented] (SPARK-20169) Groupby Bug with Sparksql

2017-04-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972300#comment-15972300 ] Takeshi Yamamuro commented on SPARK-20169: -- oh, ... good work. > Groupby Bug with Sparksql >

[jira] [Comment Edited] (SPARK-20169) Groupby Bug with Sparksql

2017-04-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972270#comment-15972270 ] Hyukjin Kwon edited comment on SPARK-20169 at 4/18/17 7:50 AM: --- Yea, I was

[jira] [Commented] (SPARK-20169) Groupby Bug with Sparksql

2017-04-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972270#comment-15972270 ] Hyukjin Kwon commented on SPARK-20169: -- Yea, I was confused too when I tried to reproduce this

[jira] [Updated] (SPARK-20366) Fix recursive join reordering: inside joins are not reordered

2017-04-18 Thread Zhenhua Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-20366: - Description: If a plan has multi-level successive joins, e.g.: {noformat} Join

[jira] [Commented] (SPARK-20286) dynamicAllocation.executorIdleTimeout is ignored after unpersist

2017-04-18 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-20286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972253#comment-15972253 ] Miguel Pérez commented on SPARK-20286: -- Thank you! I'll check it again and close the issue if I

[jira] [Commented] (SPARK-20320) AnalysisException: Columns of grouping_id (count(value#17L)) does not match grouping columns (count(value#17L))

2017-04-18 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972249#comment-15972249 ] Jacek Laskowski commented on SPARK-20320: - I'm playing with Spark SQL and multi-dimensional

[jira] [Updated] (SPARK-20366) Fix recursive join reordering: inside joins are not reordered

2017-04-18 Thread Zhenhua Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-20366: - Description: If a plan has multi-level successive joins, e.g.: Join / \

[jira] [Updated] (SPARK-20366) Fix recursive join reordering: inside joins are not reordered

2017-04-18 Thread Zhenhua Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-20366: - Description: If a plan has multi-level successive joins, e.g.: ``` Join /

[jira] [Assigned] (SPARK-20366) Fix recursive join reordering: inside joins are not reordered

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20366: Assignee: (was: Apache Spark) > Fix recursive join reordering: inside joins are not

[jira] [Commented] (SPARK-20366) Fix recursive join reordering: inside joins are not reordered

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972243#comment-15972243 ] Apache Spark commented on SPARK-20366: -- User 'wzhfy' has created a pull request for this issue:

[jira] [Assigned] (SPARK-20366) Fix recursive join reordering: inside joins are not reordered

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20366: Assignee: Apache Spark > Fix recursive join reordering: inside joins are not reordered >

[jira] [Created] (SPARK-20366) Fix recursive join reordering: inside joins are not reordered

2017-04-18 Thread Zhenhua Wang (JIRA)
Zhenhua Wang created SPARK-20366: Summary: Fix recursive join reordering: inside joins are not reordered Key: SPARK-20366 URL: https://issues.apache.org/jira/browse/SPARK-20366 Project: Spark

[jira] [Updated] (SPARK-20365) Not so accurate classpath format for AM and Containers

2017-04-18 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-20365: Summary: Not so accurate classpath format for AM and Containers (was: Inaccurate classpath format

[jira] [Commented] (SPARK-20286) dynamicAllocation.executorIdleTimeout is ignored after unpersist

2017-04-18 Thread Umesh Chaudhary (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972231#comment-15972231 ] Umesh Chaudhary commented on SPARK-20286: - Yep, +1 to the UI changes. However, I tested the

[jira] [Created] (SPARK-20365) Inaccurate classpath format for AM and Containers

2017-04-18 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-20365: --- Summary: Inaccurate classpath format for AM and Containers Key: SPARK-20365 URL: https://issues.apache.org/jira/browse/SPARK-20365 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-20174) Analyzer gives mysterious AnalysisException when posexplode used in withColumn

2017-04-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972211#comment-15972211 ] Takeshi Yamamuro edited comment on SPARK-20174 at 4/18/17 6:54 AM: --- To

[jira] [Commented] (SPARK-20174) Analyzer gives mysterious AnalysisException when posexplode used in withColumn

2017-04-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972211#comment-15972211 ] Takeshi Yamamuro commented on SPARK-20174: -- To fix this, it seems to be okay to accept

[jira] [Commented] (SPARK-20364) Parquet predicate pushdown on columns with dots return empty results

2017-04-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972171#comment-15972171 ] Hyukjin Kwon commented on SPARK-20364: -- [~aash], [~robert3005] who found this issue in

[jira] [Created] (SPARK-20364) Parquet predicate pushdown on columns with dots return empty results

2017-04-18 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-20364: Summary: Parquet predicate pushdown on columns with dots return empty results Key: SPARK-20364 URL: https://issues.apache.org/jira/browse/SPARK-20364 Project: Spark

<    1   2