[jira] [Commented] (SPARK-20367) Spark silently escapes partition column names

2017-04-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972871#comment-15972871 ] Hyukjin Kwon commented on SPARK-20367: -- Doh. I rushed reading ... > Spark silently escapes

[jira] [Created] (SPARK-20368) Support Sentry on PySpark workers

2017-04-18 Thread Alexander Shorin (JIRA)
Alexander Shorin created SPARK-20368: Summary: Support Sentry on PySpark workers Key: SPARK-20368 URL: https://issues.apache.org/jira/browse/SPARK-20368 Project: Spark Issue Type: New

[jira] [Commented] (SPARK-20367) Spark silently escapes partition column names

2017-04-18 Thread Juliusz Sompolski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972867#comment-15972867 ] Juliusz Sompolski commented on SPARK-20367: --- Hi [~hyukjin.kwon]. I tested also with parquet,

[jira] [Commented] (SPARK-15816) SQL server based on Postgres protocol

2017-04-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972971#comment-15972971 ] Takeshi Yamamuro commented on SPARK-15816: -- I put the prototype a little forward in my repo:

[jira] [Commented] (SPARK-20367) Spark silently escapes partition column names

2017-04-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972865#comment-15972865 ] Hyukjin Kwon commented on SPARK-20367: -- Actually, I did while trying to reproduce this :) {code}

[jira] [Created] (SPARK-20369) pyspark: Dynamic configuration with SparkConf does not work

2017-04-18 Thread Matthew McClain (JIRA)
Matthew McClain created SPARK-20369: --- Summary: pyspark: Dynamic configuration with SparkConf does not work Key: SPARK-20369 URL: https://issues.apache.org/jira/browse/SPARK-20369 Project: Spark

[jira] [Commented] (SPARK-20368) Support Sentry on PySpark workers

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972914#comment-15972914 ] Apache Spark commented on SPARK-20368: -- User 'kxepal' has created a pull request for this issue:

[jira] [Assigned] (SPARK-20368) Support Sentry on PySpark workers

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20368: Assignee: (was: Apache Spark) > Support Sentry on PySpark workers >

[jira] [Assigned] (SPARK-20368) Support Sentry on PySpark workers

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20368: Assignee: Apache Spark > Support Sentry on PySpark workers >

[jira] [Created] (SPARK-20370) create external table on read only location fails

2017-04-18 Thread Gaurav Shah (JIRA)
Gaurav Shah created SPARK-20370: --- Summary: create external table on read only location fails Key: SPARK-20370 URL: https://issues.apache.org/jira/browse/SPARK-20370 Project: Spark Issue Type:

[jira] [Commented] (SPARK-20364) Parquet predicate pushdown on columns with dots return empty results

2017-04-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972171#comment-15972171 ] Hyukjin Kwon commented on SPARK-20364: -- [~aash], [~robert3005] who found this issue in

[jira] [Created] (SPARK-20364) Parquet predicate pushdown on columns with dots return empty results

2017-04-18 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-20364: Summary: Parquet predicate pushdown on columns with dots return empty results Key: SPARK-20364 URL: https://issues.apache.org/jira/browse/SPARK-20364 Project: Spark

[jira] [Comment Edited] (SPARK-20174) Analyzer gives mysterious AnalysisException when posexplode used in withColumn

2017-04-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972211#comment-15972211 ] Takeshi Yamamuro edited comment on SPARK-20174 at 4/18/17 6:54 AM: --- To

[jira] [Commented] (SPARK-20174) Analyzer gives mysterious AnalysisException when posexplode used in withColumn

2017-04-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972211#comment-15972211 ] Takeshi Yamamuro commented on SPARK-20174: -- To fix this, it seems to be okay to accept

[jira] [Commented] (SPARK-6509) MDLP discretizer

2017-04-18 Thread Barry Becker (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972744#comment-15972744 ] Barry Becker commented on SPARK-6509: - As further proof of relevance, I will be giving a

[jira] [Commented] (SPARK-20356) Spark sql group by returns incorrect results after join + distinct transformations

2017-04-18 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972753#comment-15972753 ] Herman van Hovell commented on SPARK-20356: --- Here is a reproduction in scala: {noformat} val

[jira] [Commented] (SPARK-20281) Table-valued function range in SQL should use the same number of partitions as spark.range

2017-04-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972756#comment-15972756 ] Takeshi Yamamuro commented on SPARK-20281: -- IIUC they internally use the same value (that is,

[jira] [Commented] (SPARK-20356) Spark sql group by returns incorrect results after join + distinct transformations

2017-04-18 Thread Job Evers-Meltzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973100#comment-15973100 ] Job Evers-Meltzer commented on SPARK-20356: --- Commit 5ed397baa758c29c54a853d3f8fee0ad44e97c14

[jira] [Created] (SPARK-20371) R wrappers for collect_list and collect_set

2017-04-18 Thread Maciej Szymkiewicz (JIRA)
Maciej Szymkiewicz created SPARK-20371: -- Summary: R wrappers for collect_list and collect_set Key: SPARK-20371 URL: https://issues.apache.org/jira/browse/SPARK-20371 Project: Spark

[jira] [Resolved] (SPARK-20354) When I request access to the 'http: //ip:port/api/v1/applications' link, return 'sparkUser' is empty in REST API.

2017-04-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-20354. Resolution: Fixed Assignee: guoxiaolongzte Fix Version/s: 2.2.0 > When I

[jira] [Comment Edited] (SPARK-20263) create empty dataframes in sparkR

2017-04-18 Thread Ott Toomet (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973157#comment-15973157 ] Ott Toomet edited comment on SPARK-20263 at 4/18/17 5:52 PM: - for instance,

[jira] [Comment Edited] (SPARK-20263) create empty dataframes in sparkR

2017-04-18 Thread Ott Toomet (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973157#comment-15973157 ] Ott Toomet edited comment on SPARK-20263 at 4/18/17 5:51 PM: - for instance,

[jira] [Updated] (SPARK-20370) create external table on read only location fails

2017-04-18 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20370: - Component/s: (was: Spark Core) SQL > create external table on read only

[jira] [Assigned] (SPARK-20371) R wrappers for collect_list and collect_set

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20371: Assignee: (was: Apache Spark) > R wrappers for collect_list and collect_set >

[jira] [Assigned] (SPARK-20371) R wrappers for collect_list and collect_set

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20371: Assignee: Apache Spark > R wrappers for collect_list and collect_set >

[jira] [Updated] (SPARK-20367) Spark silently escapes partition column names

2017-04-18 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20367: - Component/s: (was: Spark Core) SQL > Spark silently escapes partition

[jira] [Commented] (SPARK-20371) R wrappers for collect_list and collect_set

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973165#comment-15973165 ] Apache Spark commented on SPARK-20371: -- User 'zero323' has created a pull request for this issue:

[jira] [Updated] (SPARK-20371) R wrappers for collect_list and collect_set

2017-04-18 Thread Maciej Szymkiewicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-20371: --- Affects Version/s: 2.0.0 > R wrappers for collect_list and collect_set >

[jira] [Assigned] (SPARK-20372) Word2Vec Continuous Bag Of Words model

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20372: Assignee: (was: Apache Spark) > Word2Vec Continuous Bag Of Words model >

[jira] [Commented] (SPARK-20372) Word2Vec Continuous Bag Of Words model

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973187#comment-15973187 ] Apache Spark commented on SPARK-20372: -- User 'shubhamchopra' has created a pull request for this

[jira] [Assigned] (SPARK-20372) Word2Vec Continuous Bag Of Words model

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20372: Assignee: Apache Spark > Word2Vec Continuous Bag Of Words model >

[jira] [Commented] (SPARK-20341) Support BigIngeger values > 19 precision

2017-04-18 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973227#comment-15973227 ] Kazuaki Ishizaki commented on SPARK-20341: -- Master branch also throws an exception. > Support

[jira] [Commented] (SPARK-20263) create empty dataframes in sparkR

2017-04-18 Thread Ott Toomet (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973157#comment-15973157 ] Ott Toomet commented on SPARK-20263: for instance, by rbinding in a loop. I have several pieces of

[jira] [Created] (SPARK-20372) Word2Vec Continuous Bag Of Words model

2017-04-18 Thread Shubham Chopra (JIRA)
Shubham Chopra created SPARK-20372: -- Summary: Word2Vec Continuous Bag Of Words model Key: SPARK-20372 URL: https://issues.apache.org/jira/browse/SPARK-20372 Project: Spark Issue Type:

[jira] [Updated] (SPARK-19968) Use a cached instance of KafkaProducer for writing to kafka via KafkaSink.

2017-04-18 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19968: - Target Version/s: 2.3.0 (was: 2.2.0) > Use a cached instance of KafkaProducer for writing to

[jira] [Commented] (SPARK-20286) dynamicAllocation.executorIdleTimeout is ignored after unpersist

2017-04-18 Thread Umesh Chaudhary (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972231#comment-15972231 ] Umesh Chaudhary commented on SPARK-20286: - Yep, +1 to the UI changes. However, I tested the

[jira] [Updated] (SPARK-20365) Not so accurate classpath format for AM and Containers

2017-04-18 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-20365: Summary: Not so accurate classpath format for AM and Containers (was: Inaccurate classpath format

[jira] [Updated] (SPARK-20366) Fix recursive join reordering: inside joins are not reordered

2017-04-18 Thread Zhenhua Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-20366: - Description: If a plan has multi-level successive joins, e.g.: Join / \

[jira] [Commented] (SPARK-20320) AnalysisException: Columns of grouping_id (count(value#17L)) does not match grouping columns (count(value#17L))

2017-04-18 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972249#comment-15972249 ] Jacek Laskowski commented on SPARK-20320: - I'm playing with Spark SQL and multi-dimensional

[jira] [Created] (SPARK-20366) Fix recursive join reordering: inside joins are not reordered

2017-04-18 Thread Zhenhua Wang (JIRA)
Zhenhua Wang created SPARK-20366: Summary: Fix recursive join reordering: inside joins are not reordered Key: SPARK-20366 URL: https://issues.apache.org/jira/browse/SPARK-20366 Project: Spark

[jira] [Updated] (SPARK-20366) Fix recursive join reordering: inside joins are not reordered

2017-04-18 Thread Zhenhua Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-20366: - Description: If a plan has multi-level successive joins, e.g.: ``` Join /

[jira] [Commented] (SPARK-20286) dynamicAllocation.executorIdleTimeout is ignored after unpersist

2017-04-18 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-20286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972253#comment-15972253 ] Miguel Pérez commented on SPARK-20286: -- Thank you! I'll check it again and close the issue if I

[jira] [Comment Edited] (SPARK-20169) Groupby Bug with Sparksql

2017-04-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972270#comment-15972270 ] Hyukjin Kwon edited comment on SPARK-20169 at 4/18/17 7:50 AM: --- Yea, I was

[jira] [Created] (SPARK-20365) Inaccurate classpath format for AM and Containers

2017-04-18 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-20365: --- Summary: Inaccurate classpath format for AM and Containers Key: SPARK-20365 URL: https://issues.apache.org/jira/browse/SPARK-20365 Project: Spark Issue Type:

[jira] [Commented] (SPARK-20366) Fix recursive join reordering: inside joins are not reordered

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972243#comment-15972243 ] Apache Spark commented on SPARK-20366: -- User 'wzhfy' has created a pull request for this issue:

[jira] [Assigned] (SPARK-20366) Fix recursive join reordering: inside joins are not reordered

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20366: Assignee: (was: Apache Spark) > Fix recursive join reordering: inside joins are not

[jira] [Assigned] (SPARK-20366) Fix recursive join reordering: inside joins are not reordered

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20366: Assignee: Apache Spark > Fix recursive join reordering: inside joins are not reordered >

[jira] [Updated] (SPARK-20366) Fix recursive join reordering: inside joins are not reordered

2017-04-18 Thread Zhenhua Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-20366: - Description: If a plan has multi-level successive joins, e.g.: {noformat} Join

[jira] [Commented] (SPARK-20169) Groupby Bug with Sparksql

2017-04-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972270#comment-15972270 ] Hyukjin Kwon commented on SPARK-20169: -- Yea, I was confused too when I tried to reproduce this

[jira] [Commented] (SPARK-20169) Groupby Bug with Sparksql

2017-04-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972300#comment-15972300 ] Takeshi Yamamuro commented on SPARK-20169: -- oh, ... good work. > Groupby Bug with Sparksql >

[jira] [Commented] (SPARK-6509) MDLP discretizer

2017-04-18 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972806#comment-15972806 ] Sergio Ramírez commented on SPARK-6509: --- Thanks again Barry for your support. I hope this proof can

[jira] [Commented] (SPARK-20356) Spark sql group by returns incorrect results after join + distinct transformations

2017-04-18 Thread Ed Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972544#comment-15972544 ] Ed Lee commented on SPARK-20356: really quite dangerous bug > Spark sql group by returns incorrect

[jira] [Assigned] (SPARK-20366) Fix recursive join reordering: inside joins are not reordered

2017-04-18 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-20366: --- Assignee: Zhenhua Wang > Fix recursive join reordering: inside joins are not reordered >

[jira] [Resolved] (SPARK-20366) Fix recursive join reordering: inside joins are not reordered

2017-04-18 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-20366. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17668

[jira] [Assigned] (SPARK-20281) Table-valued function range in SQL should use the same number of partitions as spark.range

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20281: Assignee: Apache Spark > Table-valued function range in SQL should use the same number of

[jira] [Assigned] (SPARK-20281) Table-valued function range in SQL should use the same number of partitions as spark.range

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20281: Assignee: (was: Apache Spark) > Table-valued function range in SQL should use the

[jira] [Commented] (SPARK-20281) Table-valued function range in SQL should use the same number of partitions as spark.range

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972758#comment-15972758 ] Apache Spark commented on SPARK-20281: -- User 'maropu' has created a pull request for this issue:

[jira] [Created] (SPARK-20367) Spark silently escapes partition column names

2017-04-18 Thread Juliusz Sompolski (JIRA)
Juliusz Sompolski created SPARK-20367: - Summary: Spark silently escapes partition column names Key: SPARK-20367 URL: https://issues.apache.org/jira/browse/SPARK-20367 Project: Spark

[jira] [Commented] (SPARK-20343) SBT master build for Hadoop 2.6 in Jenkins fails due to Avro version resolution

2017-04-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972842#comment-15972842 ] Hyukjin Kwon commented on SPARK-20343: -- Please let me know if anyone is able to reproduce this. I am

[jira] [Commented] (SPARK-20367) Spark silently escapes partition column names

2017-04-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972850#comment-15972850 ] Hyukjin Kwon commented on SPARK-20367: -- I guess probably this is not a CSV datasource specific

[jira] [Assigned] (SPARK-20377) Fix JavaStructuredSessionization example

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20377: Assignee: Tathagata Das (was: Apache Spark) > Fix JavaStructuredSessionization example >

[jira] [Commented] (SPARK-20341) Support BigIngeger values > 19 precision

2017-04-18 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973343#comment-15973343 ] Kazuaki Ishizaki commented on SPARK-20341: -- This exception is thrown since {{BigInt}} and

[jira] [Created] (SPARK-20374) Encoder generated using Java beans causes corruption in MapGroupsWithState

2017-04-18 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-20374: - Summary: Encoder generated using Java beans causes corruption in MapGroupsWithState Key: SPARK-20374 URL: https://issues.apache.org/jira/browse/SPARK-20374

[jira] [Commented] (SPARK-20015) Document R Structured Streaming (experimental) in R vignettes and R & SS programming guide

2017-04-18 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973461#comment-15973461 ] Joseph K. Bradley commented on SPARK-20015: --- [~felixcheung] I see several R issues listed as

[jira] [Commented] (SPARK-20036) impossible to read a whole kafka topic using kafka 0.10 and spark 2.0.0

2017-04-18 Thread Ryan Stack (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973497#comment-15973497 ] Ryan Stack commented on SPARK-20036: I had the same issue. Removing the explicit dependencies in my

[jira] [Updated] (SPARK-12297) Add work-around for Parquet/Hive int96 timestamp bug.

2017-04-18 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-12297: - Description: Spark copied Hive's behavior for parquet, but this was inconsistent with other

[jira] [Closed] (SPARK-20374) Encoder generated using Java beans causes corruption in MapGroupsWithState

2017-04-18 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das closed SPARK-20374. - Resolution: Duplicate > Encoder generated using Java beans causes corruption in

[jira] [Created] (SPARK-20376) Make StateStoreProvider plugable

2017-04-18 Thread Yogesh Mahajan (JIRA)
Yogesh Mahajan created SPARK-20376: -- Summary: Make StateStoreProvider plugable Key: SPARK-20376 URL: https://issues.apache.org/jira/browse/SPARK-20376 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-20374) Encoder generated using Java beans causes corruption in MapGroupsWithState

2017-04-18 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-20374: -- Description: Running the example

[jira] [Commented] (SPARK-20036) impossible to read a whole kafka topic using kafka 0.10 and spark 2.0.0

2017-04-18 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973507#comment-15973507 ] Cody Koeninger commented on SPARK-20036: I'll submit a PR to add a note to the docs about this.

[jira] [Assigned] (SPARK-20036) impossible to read a whole kafka topic using kafka 0.10 and spark 2.0.0

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20036: Assignee: (was: Apache Spark) > impossible to read a whole kafka topic using kafka

[jira] [Commented] (SPARK-20036) impossible to read a whole kafka topic using kafka 0.10 and spark 2.0.0

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973538#comment-15973538 ] Apache Spark commented on SPARK-20036: -- User 'koeninger' has created a pull request for this issue:

[jira] [Assigned] (SPARK-20036) impossible to read a whole kafka topic using kafka 0.10 and spark 2.0.0

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20036: Assignee: Apache Spark > impossible to read a whole kafka topic using kafka 0.10 and

[jira] [Created] (SPARK-20377) Fix JavaStructuredSessionization example

2017-04-18 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-20377: - Summary: Fix JavaStructuredSessionization example Key: SPARK-20377 URL: https://issues.apache.org/jira/browse/SPARK-20377 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-20379) Allow setting SSL-related passwords through env variables

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973633#comment-15973633 ] Apache Spark commented on SPARK-20379: -- User 'vanzin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-20379) Allow setting SSL-related passwords through env variables

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20379: Assignee: (was: Apache Spark) > Allow setting SSL-related passwords through env

[jira] [Updated] (SPARK-20377) Fix JavaStructuredSessionization example

2017-04-18 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-20377: -- Description: Extra accessors in java bean class causes incorrect encoder generation. This led

[jira] [Commented] (SPARK-20377) Fix JavaStructuredSessionization example

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973594#comment-15973594 ] Apache Spark commented on SPARK-20377: -- User 'tdas' has created a pull request for this issue:

[jira] [Assigned] (SPARK-20377) Fix JavaStructuredSessionization example

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20377: Assignee: Apache Spark (was: Tathagata Das) > Fix JavaStructuredSessionization example >

[jira] [Assigned] (SPARK-20360) Create repr functions for interpreters to use

2017-04-18 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-20360: --- Assignee: Kyle Kelley > Create repr functions for interpreters to use >

[jira] [Assigned] (SPARK-20375) R wrappers for array and map

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20375: Assignee: Apache Spark > R wrappers for array and map > > >

[jira] [Created] (SPARK-20378) StreamSinkProvider should provide schema in createSink.

2017-04-18 Thread Yogesh Mahajan (JIRA)
Yogesh Mahajan created SPARK-20378: -- Summary: StreamSinkProvider should provide schema in createSink. Key: SPARK-20378 URL: https://issues.apache.org/jira/browse/SPARK-20378 Project: Spark

[jira] [Updated] (SPARK-20378) StreamSinkProvider should provide schema in createSink.

2017-04-18 Thread Yogesh Mahajan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yogesh Mahajan updated SPARK-20378: --- Description: We have our own Sink implementation based on our in memory store and this sink

[jira] [Assigned] (SPARK-20379) Allow setting SSL-related passwords through env variables

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20379: Assignee: Apache Spark > Allow setting SSL-related passwords through env variables >

[jira] [Resolved] (SPARK-20360) Create repr functions for interpreters to use

2017-04-18 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-20360. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17662

[jira] [Created] (SPARK-20373) Batch queries with 'Dataset/DataFrame.withWatermark()` does not execute

2017-04-18 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-20373: - Summary: Batch queries with 'Dataset/DataFrame.withWatermark()` does not execute Key: SPARK-20373 URL: https://issues.apache.org/jira/browse/SPARK-20373 Project:

[jira] [Comment Edited] (SPARK-20356) Spark sql group by returns incorrect results after join + distinct transformations

2017-04-18 Thread Dilip Biswal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973455#comment-15973455 ] Dilip Biswal edited comment on SPARK-20356 at 4/18/17 8:47 PM: --- [~viirya]

[jira] [Comment Edited] (SPARK-20356) Spark sql group by returns incorrect results after join + distinct transformations

2017-04-18 Thread Dilip Biswal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973455#comment-15973455 ] Dilip Biswal edited comment on SPARK-20356 at 4/18/17 8:46 PM: --- [~viirya]

[jira] [Assigned] (SPARK-20375) R wrappers for array and map

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20375: Assignee: (was: Apache Spark) > R wrappers for array and map >

[jira] [Commented] (SPARK-20375) R wrappers for array and map

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973524#comment-15973524 ] Apache Spark commented on SPARK-20375: -- User 'zero323' has created a pull request for this issue:

[jira] [Commented] (SPARK-20153) Support Multiple aws credentials in order to access multiple Hive on S3 table in spark application

2017-04-18 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973541#comment-15973541 ] Steve Loughran commented on SPARK-20153: [~tafra...@gmail.com] : thanks for discovering that. I

[jira] [Commented] (SPARK-20036) impossible to read a whole kafka topic using kafka 0.10 and spark 2.0.0

2017-04-18 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973554#comment-15973554 ] Cody Koeninger commented on SPARK-20036: [~danielnuriyev] what actually happened when you removed

[jira] [Created] (SPARK-20379) Allow setting SSL-related passwords through env variables

2017-04-18 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-20379: -- Summary: Allow setting SSL-related passwords through env variables Key: SPARK-20379 URL: https://issues.apache.org/jira/browse/SPARK-20379 Project: Spark

[jira] [Resolved] (SPARK-20344) Duplicate call in FairSchedulableBuilder.addTaskSetManager

2017-04-18 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-20344. --- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17647

[jira] [Commented] (SPARK-20174) Analyzer gives mysterious AnalysisException when posexplode used in withColumn

2017-04-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972359#comment-15972359 ] Takeshi Yamamuro commented on SPARK-20174: -- You could fix this like

[jira] [Comment Edited] (SPARK-20174) Analyzer gives mysterious AnalysisException when posexplode used in withColumn

2017-04-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972359#comment-15972359 ] Takeshi Yamamuro edited comment on SPARK-20174 at 4/18/17 9:09 AM: --- You

[jira] [Assigned] (SPARK-20344) Duplicate call in FairSchedulableBuilder.addTaskSetManager

2017-04-18 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-20344: - Assignee: Robert Stupp > Duplicate call in FairSchedulableBuilder.addTaskSetManager >

[jira] [Commented] (SPARK-19995) Using real user to connect HiveMetastore in HiveClientImpl

2017-04-18 Thread meiyoula (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972373#comment-15972373 ] meiyoula commented on SPARK-19995: -- Will the token be expired? > Using real user to connect

[jira] [Commented] (SPARK-20364) Parquet predicate pushdown on columns with dots return empty results

2017-04-18 Thread Robert Kruszewski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972400#comment-15972400 ] Robert Kruszewski commented on SPARK-20364: --- Looks like parquet doesn't differentiate between

[jira] [Comment Edited] (SPARK-20364) Parquet predicate pushdown on columns with dots return empty results

2017-04-18 Thread Robert Kruszewski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972400#comment-15972400 ] Robert Kruszewski edited comment on SPARK-20364 at 4/18/17 9:36 AM:

  1   2   >