[jira] [Assigned] (SPARK-19956) Optimize a location order of blocks with topology information

2017-03-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19956: Assignee: (was: Apache Spark) > Optimize a location order of blocks with topology

[jira] [Assigned] (SPARK-19956) Optimize a location order of blocks with topology information

2017-03-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19956: Assignee: Apache Spark > Optimize a location order of blocks with topology information >

[jira] [Created] (SPARK-19956) Optimize a location order of blocks with topology information

2017-03-14 Thread coneyliu (JIRA)
coneyliu created SPARK-19956: Summary: Optimize a location order of blocks with topology information Key: SPARK-19956 URL: https://issues.apache.org/jira/browse/SPARK-19956 Project: Spark Issue

[jira] [Resolved] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

2017-03-14 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-18112. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17232

[jira] [Resolved] (SPARK-19828) R to support JSON array in column from_json

2017-03-14 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-19828. -- Resolution: Fixed Assignee: Hyukjin Kwon Fix Version/s: 2.2.0

[jira] [Resolved] (SPARK-19887) __HIVE_DEFAULT_PARTITION__ is not interpreted as NULL partition value in partitioned persisted tables

2017-03-14 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-19887. - Resolution: Fixed Assignee: Wenchen Fan Fix Version/s: 2.2.0

[jira] [Updated] (SPARK-19817) make it clear that `timeZone` option is a general option in DataFrameReader/Writer, DataStreamReader/Writer

2017-03-14 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liwei Lin updated SPARK-19817: -- Summary: make it clear that `timeZone` option is a general option in DataFrameReader/Writer,

[jira] [Updated] (SPARK-19817) make it clear that `timeZone` option is a general option in DataFrameReader/Writer

2017-03-14 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liwei Lin updated SPARK-19817: -- Component/s: Structured Streaming > make it clear that `timeZone` option is a general option in >

[jira] [Resolved] (SPARK-19918) Use TextFileFormat in implementation of JsonFileFormat

2017-03-14 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-19918. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17255

[jira] [Assigned] (SPARK-19918) Use TextFileFormat in implementation of JsonFileFormat

2017-03-14 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-19918: --- Assignee: Hyukjin Kwon > Use TextFileFormat in implementation of JsonFileFormat >

[jira] [Closed] (SPARK-19881) Support Dynamic Partition Inserts params with SET command

2017-03-14 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-19881. - > Support Dynamic Partition Inserts params with SET command >

[jira] [Closed] (SPARK-17277) Set hive conf failed

2017-03-14 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-17277. - > Set hive conf failed > > > Key: SPARK-17277 >

[jira] [Resolved] (SPARK-17277) Set hive conf failed

2017-03-14 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-17277. --- Resolution: Won't Fix As mentioned in the related issue

[jira] [Resolved] (SPARK-19881) Support Dynamic Partition Inserts params with SET command

2017-03-14 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-19881. --- Resolution: Won't Fix As mentioned in

[jira] [Commented] (SPARK-17465) Inappropriate memory management in `org.apache.spark.storage.MemoryStore` may lead to memory leak

2017-03-14 Thread agate (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925394#comment-15925394 ] agate commented on SPARK-17465: --- [~saturday_s] Thanks for this fix! It really helped us. We were using

[jira] [Closed] (SPARK-19834) csv escape of quote escape

2017-03-14 Thread Soonmok Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Soonmok Kwon closed SPARK-19834. Resolution: Later Closed for now. Wil be re-open when spark uses uniVocity-parser 2.4.0+. > csv

[jira] [Comment Edited] (SPARK-19834) csv escape of quote escape

2017-03-14 Thread Soonmok Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925380#comment-15925380 ] Soonmok Kwon edited comment on SPARK-19834 at 3/15/17 1:25 AM: --- Agreed

[jira] [Commented] (SPARK-19834) csv escape of quote escape

2017-03-14 Thread Soonmok Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925380#comment-15925380 ] Soonmok Kwon commented on SPARK-19834: -- To resolve this issue we need to enable uniVocity csv parser

[jira] [Commented] (SPARK-19834) csv escape of quote escape

2017-03-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925375#comment-15925375 ] Hyukjin Kwon commented on SPARK-19834: -- Just for other guys to easily track this, I guess this is a

[jira] [Updated] (SPARK-19834) csv escape of quote escape

2017-03-14 Thread Soonmok Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Soonmok Kwon updated SPARK-19834: - Summary: csv escape of quote escape (was: csv encoding/decoding error not using escape of

[jira] [Commented] (SPARK-19950) nullable ignored when df.load() is executed for file-based data source

2017-03-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925367#comment-15925367 ] Hyukjin Kwon commented on SPARK-19950: -- Yes. Just to help, up to my knowledge, currently, the

[jira] [Commented] (SPARK-19950) nullable ignored when df.load() is executed for file-based data source

2017-03-14 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925357#comment-15925357 ] Kazuaki Ishizaki commented on SPARK-19950: -- [~hyukjin.kwon] Thank you for pointing out

[jira] [Reopened] (SPARK-19803) Flaky BlockManagerProactiveReplicationSuite tests

2017-03-14 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kay Ousterhout reopened SPARK-19803: Assignee: Shubham Chopra (was: Genmao Yu) > Flaky BlockManagerProactiveReplicationSuite

[jira] [Commented] (SPARK-19803) Flaky BlockManagerProactiveReplicationSuite tests

2017-03-14 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925353#comment-15925353 ] Kay Ousterhout commented on SPARK-19803: This does not appear to be fixed -- it looks like

[jira] [Comment Edited] (SPARK-19875) Map->filter on many columns gets stuck in constraint inference optimization code

2017-03-14 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925332#comment-15925332 ] Takeshi Yamamuro edited comment on SPARK-19875 at 3/15/17 12:32 AM:

[jira] [Commented] (SPARK-19875) Map->filter on many columns gets stuck in constraint inference optimization code

2017-03-14 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925332#comment-15925332 ] Takeshi Yamamuro commented on SPARK-19875: -- Hi, Sameer. If you understand a concrete reason

[jira] [Commented] (SPARK-19954) Joining to a unioned DataFrame does not produce expected result.

2017-03-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925310#comment-15925310 ] Hyukjin Kwon commented on SPARK-19954: -- Is this a blocker BTW? {quote} pointless to release without

[jira] [Updated] (SPARK-16472) Inconsistent nullability in schema after being read

2017-03-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-16472: - Summary: Inconsistent nullability in schema after being read (was: Inconsistent nullability in

[jira] [Comment Edited] (SPARK-19950) nullable ignored when df.load() is executed for file-based data source

2017-03-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925301#comment-15925301 ] Hyukjin Kwon edited comment on SPARK-19950 at 3/15/17 12:04 AM: [~kiszk],

[jira] [Commented] (SPARK-19950) nullable ignored when df.load() is executed for file-based data source

2017-03-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925301#comment-15925301 ] Hyukjin Kwon commented on SPARK-19950: -- [~kiszk], do you think we maybe resolve this JIRA as a

[jira] [Commented] (SPARK-19955) Update run-tests to support conda

2017-03-14 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925295#comment-15925295 ] holdenk commented on SPARK-19955: - It appears that the current Jenkin workers have conda installed & an

[jira] [Updated] (SPARK-19955) Update run-tests to support conda

2017-03-14 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk updated SPARK-19955: Issue Type: Sub-task (was: Improvement) Parent: SPARK-12661 > Update run-tests to support conda >

[jira] [Created] (SPARK-19955) Update run-tests to support conda

2017-03-14 Thread holdenk (JIRA)
holdenk created SPARK-19955: --- Summary: Update run-tests to support conda Key: SPARK-19955 URL: https://issues.apache.org/jira/browse/SPARK-19955 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-19094) Plumb through logging/error messages from the JVM to Jupyter PySpark

2017-03-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19094: Assignee: Apache Spark > Plumb through logging/error messages from the JVM to Jupyter

[jira] [Commented] (SPARK-19094) Plumb through logging/error messages from the JVM to Jupyter PySpark

2017-03-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925285#comment-15925285 ] Apache Spark commented on SPARK-19094: -- User 'holdenk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19094) Plumb through logging/error messages from the JVM to Jupyter PySpark

2017-03-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19094: Assignee: (was: Apache Spark) > Plumb through logging/error messages from the JVM to

[jira] [Commented] (SPARK-19282) RandomForestRegressionModel summary should expose getMaxDepth

2017-03-14 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925159#comment-15925159 ] Bryan Cutler commented on SPARK-19282: -- [~iamshrek] I am currently working on SPARK-10931 and I'll

[jira] [Resolved] (SPARK-19629) Partitioning of Parquet is not considered correctly at loading in local[X] mode

2017-03-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-19629. -- Resolution: Not A Problem {quote} What could other solutions be, if this is not a bug? {quote}

[jira] [Updated] (SPARK-19954) Joining to a unioned DataFrame does not produce expected result.

2017-03-14 Thread Arun Allamsetty (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Allamsetty updated SPARK-19954: Description: I found this bug while trying to update from Spark 1.6.1 to 2.1.0. The bug is

[jira] [Commented] (SPARK-18603) Support `OuterReference` in projection list of IN correlated subqueries

2017-03-14 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925140#comment-15925140 ] Dongjoon Hyun commented on SPARK-18603: --- Thank YOU! > Support `OuterReference` in projection list

[jira] [Updated] (SPARK-19954) Joining to a unioned DataFrame does not produce expected result.

2017-03-14 Thread Arun Allamsetty (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Allamsetty updated SPARK-19954: Description: I found this bug while trying to update from Spark 1.6.1 to 2.1.0. The bug is

[jira] [Created] (SPARK-19954) Joining to a unioned DataFrame does not produce expected result.

2017-03-14 Thread Arun Allamsetty (JIRA)
Arun Allamsetty created SPARK-19954: --- Summary: Joining to a unioned DataFrame does not produce expected result. Key: SPARK-19954 URL: https://issues.apache.org/jira/browse/SPARK-19954 Project:

[jira] [Commented] (SPARK-14649) DagScheduler re-starts all running tasks on fetch failure

2017-03-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925125#comment-15925125 ] Apache Spark commented on SPARK-14649: -- User 'sitalkedia' has created a pull request for this issue:

[jira] [Commented] (SPARK-18603) Support `OuterReference` in projection list of IN correlated subqueries

2017-03-14 Thread Nattavut Sutyanyong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925122#comment-15925122 ] Nattavut Sutyanyong commented on SPARK-18603: - I will include it in my list and will

[jira] [Commented] (SPARK-18603) Support `OuterReference` in projection list of IN correlated subqueries

2017-03-14 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925115#comment-15925115 ] Dongjoon Hyun commented on SPARK-18603: --- For this kind of issues, I think you are the best person

[jira] [Assigned] (SPARK-19953) RandomForest Models should use the UID of Estimator when fit

2017-03-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19953: Assignee: Apache Spark > RandomForest Models should use the UID of Estimator when fit >

[jira] [Assigned] (SPARK-19953) RandomForest Models should use the UID of Estimator when fit

2017-03-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19953: Assignee: (was: Apache Spark) > RandomForest Models should use the UID of Estimator

[jira] [Commented] (SPARK-19953) RandomForest Models should use the UID of Estimator when fit

2017-03-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925107#comment-15925107 ] Apache Spark commented on SPARK-19953: -- User 'BryanCutler' has created a pull request for this

[jira] [Commented] (SPARK-18603) Support `OuterReference` in projection list of IN correlated subqueries

2017-03-14 Thread Nattavut Sutyanyong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925092#comment-15925092 ] Nattavut Sutyanyong commented on SPARK-18603: - [~dongjoon] Now that the first phase of the

[jira] [Commented] (SPARK-19953) RandomForest Models should use the UID of Estimator when fit

2017-03-14 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925088#comment-15925088 ] Bryan Cutler commented on SPARK-19953: -- I'll push the patch for this > RandomForest Models should

[jira] [Created] (SPARK-19953) RandomForest Models should use the UID of Estimator when fit

2017-03-14 Thread Bryan Cutler (JIRA)
Bryan Cutler created SPARK-19953: Summary: RandomForest Models should use the UID of Estimator when fit Key: SPARK-19953 URL: https://issues.apache.org/jira/browse/SPARK-19953 Project: Spark

[jira] [Commented] (SPARK-19556) Broadcast data is not encrypted when I/O encryption is on

2017-03-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925067#comment-15925067 ] Apache Spark commented on SPARK-19556: -- User 'vanzin' has created a pull request for this issue:

[jira] [Updated] (SPARK-16617) Upgrade to Avro 1.8.x

2017-03-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-16617: -- Affects Version/s: 2.1.0 Target Version/s: 3.0.0 Component/s: Spark Core

[jira] [Updated] (SPARK-19817) make it clear that `timeZone` option is a general option in DataFrameReader/Writer

2017-03-14 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-19817: Fix Version/s: 2.2.0 > make it clear that `timeZone` option is a general option in >

[jira] [Resolved] (SPARK-19817) make it clear that `timeZone` option is a general option in DataFrameReader/Writer

2017-03-14 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-19817. - Resolution: Fixed > make it clear that `timeZone` option is a general option in >

[jira] [Resolved] (SPARK-18966) NOT IN subquery with correlated expressions may return incorrect result

2017-03-14 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-18966. --- Resolution: Fixed Assignee: Nattavut Sutyanyong Fix Version/s: 2.2.0

[jira] [Updated] (SPARK-19952) Remove specialized catalog related analysis exceptions

2017-03-14 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-19952: Description: We introduce catalog specific analysis exceptions (that extends AnalysisException)

[jira] [Updated] (SPARK-19952) Remove specialized catalog related analysis exceptions

2017-03-14 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-19952: Description: We introduce catalog specific analysis exceptions (that extends AnalysisException) in

[jira] [Created] (SPARK-19952) Remove specialized catalog related analysis exceptions

2017-03-14 Thread Herman van Hovell (JIRA)
Herman van Hovell created SPARK-19952: - Summary: Remove specialized catalog related analysis exceptions Key: SPARK-19952 URL: https://issues.apache.org/jira/browse/SPARK-19952 Project: Spark

[jira] [Commented] (SPARK-19282) RandomForestRegressionModel summary should expose getMaxDepth

2017-03-14 Thread Xin Ren (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924728#comment-15924728 ] Xin Ren commented on SPARK-19282: - thanks Bryan, could you please create some sub tasks under

[jira] [Created] (SPARK-19951) Add string concatenate operator || to Spark SQL

2017-03-14 Thread Herman van Hovell (JIRA)
Herman van Hovell created SPARK-19951: - Summary: Add string concatenate operator || to Spark SQL Key: SPARK-19951 URL: https://issues.apache.org/jira/browse/SPARK-19951 Project: Spark

[jira] [Resolved] (SPARK-19933) TPCDS Q70 went wrong while explaining

2017-03-14 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-19933. --- Resolution: Fixed Assignee: Herman van Hovell Fix Version/s: 2.2.0 >

[jira] [Resolved] (SPARK-19923) Remove unnecessary type conversion per call in Hive

2017-03-14 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-19923. --- Resolution: Fixed Assignee: Takeshi Yamamuro Fix Version/s: 2.2.0 >

[jira] [Commented] (SPARK-19899) FPGrowth input column naming

2017-03-14 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924677#comment-15924677 ] Felix Cheung commented on SPARK-19899: -- +1 on "itemsCol" looks like it is defaulting to "items" for

[jira] [Commented] (SPARK-19416) Dataset.schema is inconsistent with Dataset in handling columns with periods

2017-03-14 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924646#comment-15924646 ] Reynold Xin commented on SPARK-19416: - We probably can't change any of them now, unless we introduce

[jira] [Commented] (SPARK-19282) RandomForestRegressionModel summary should expose getMaxDepth

2017-03-14 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924644#comment-15924644 ] Bryan Cutler commented on SPARK-19282: -- This is common issue with all PySpark ML Models and

[jira] [Assigned] (SPARK-18961) Support `SHOW TABLE EXTENDED ... PARTITION` statement

2017-03-14 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-18961: --- Assignee: Jiang Xingbo > Support `SHOW TABLE EXTENDED ... PARTITION` statement >

[jira] [Closed] (SPARK-18961) Support `SHOW TABLE EXTENDED ... PARTITION` statement

2017-03-14 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li closed SPARK-18961. --- Resolution: Fixed Fix Version/s: 2.2.0 > Support `SHOW TABLE EXTENDED ... PARTITION` statement >

[jira] [Commented] (SPARK-18966) NOT IN subquery with correlated expressions may return incorrect result

2017-03-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924593#comment-15924593 ] Apache Spark commented on SPARK-18966: -- User 'nsyca' has created a pull request for this issue:

[jira] [Assigned] (SPARK-18966) NOT IN subquery with correlated expressions may return incorrect result

2017-03-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18966: Assignee: Apache Spark > NOT IN subquery with correlated expressions may return incorrect

[jira] [Assigned] (SPARK-18966) NOT IN subquery with correlated expressions may return incorrect result

2017-03-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18966: Assignee: (was: Apache Spark) > NOT IN subquery with correlated expressions may

[jira] [Commented] (SPARK-19424) Wrong runtime type in RDD when reading from avro with custom serializer

2017-03-14 Thread Nira Amit (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924573#comment-15924573 ] Nira Amit commented on SPARK-19424: --- [~hvanhovell] The solution would be to not ignore the "Unchecked

[jira] [Closed] (SPARK-19424) Wrong runtime type in RDD when reading from avro with custom serializer

2017-03-14 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell closed SPARK-19424. - Resolution: Not A Problem > Wrong runtime type in RDD when reading from avro with custom

[jira] [Commented] (SPARK-19424) Wrong runtime type in RDD when reading from avro with custom serializer

2017-03-14 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924564#comment-15924564 ] Herman van Hovell commented on SPARK-19424: --- [~amitnira] Like Sean said, you are not proposing

[jira] [Commented] (SPARK-19424) Wrong runtime type in RDD when reading from avro with custom serializer

2017-03-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924517#comment-15924517 ] Sean Owen commented on SPARK-19424: --- All: I'm going to contact INFRA about blocking further changes. I

[jira] [Reopened] (SPARK-19424) Wrong runtime type in RDD when reading from avro with custom serializer

2017-03-14 Thread Nira Amit (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nira Amit reopened SPARK-19424: --- I managed to get my code to work, yes. The problem of the wrong type at runtime is not resolved,

[jira] [Assigned] (SPARK-19950) nullable ignored when df.load() is executed for file-based data source

2017-03-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19950: Assignee: (was: Apache Spark) > nullable ignored when df.load() is executed for

[jira] [Commented] (SPARK-19950) nullable ignored when df.load() is executed for file-based data source

2017-03-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924509#comment-15924509 ] Apache Spark commented on SPARK-19950: -- User 'kiszk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19950) nullable ignored when df.load() is executed for file-based data source

2017-03-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19950: Assignee: Apache Spark > nullable ignored when df.load() is executed for file-based data

[jira] [Created] (SPARK-19950) nullable ignored when df.load() is executed for file-based data source

2017-03-14 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-19950: Summary: nullable ignored when df.load() is executed for file-based data source Key: SPARK-19950 URL: https://issues.apache.org/jira/browse/SPARK-19950

[jira] [Resolved] (SPARK-19424) Wrong runtime type in RDD when reading from avro with custom serializer

2017-03-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19424. --- Resolution: Not A Problem You resolved the problem above though, yourself. You hadn't passed the

[jira] [Closed] (SPARK-19424) Wrong runtime type in RDD when reading from avro with custom serializer

2017-03-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen closed SPARK-19424. - > Wrong runtime type in RDD when reading from avro with custom serializer >

[jira] [Reopened] (SPARK-19424) Wrong runtime type in RDD when reading from avro with custom serializer

2017-03-14 Thread Nira Amit (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nira Amit reopened SPARK-19424: --- Sean, I am not trying to piss you off but to investigate a problem. If your explanation is that it is

[jira] [Commented] (SPARK-19899) FPGrowth input column naming

2017-03-14 Thread Maciej Szymkiewicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924435#comment-15924435 ] Maciej Szymkiewicz commented on SPARK-19899: "itemsCol" sounds good. What should we use as a

[jira] [Commented] (SPARK-11798) Datanucleus jars is missing under lib_managed/jars

2017-03-14 Thread Luca Menichetti (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924420#comment-15924420 ] Luca Menichetti commented on SPARK-11798: - I have exactly the same issue, I had to include all

[jira] [Resolved] (SPARK-19424) Wrong runtime type in RDD when reading from avro with custom serializer

2017-03-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19424. --- Resolution: Not A Problem This was even further discussed at

[jira] [Closed] (SPARK-19424) Wrong runtime type in RDD when reading from avro with custom serializer

2017-03-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen closed SPARK-19424. - > Wrong runtime type in RDD when reading from avro with custom serializer >

[jira] [Reopened] (SPARK-19424) Wrong runtime type in RDD when reading from avro with custom serializer

2017-03-14 Thread Nira Amit (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nira Amit reopened SPARK-19424: --- Re-opening this issue because throwing unexpected ClassCastExceptions is not an accepted behavior of a

[jira] [Commented] (SPARK-19282) RandomForestRegressionModel summary should expose getMaxDepth

2017-03-14 Thread Xin Ren (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924374#comment-15924374 ] Xin Ren commented on SPARK-19282: - sure, I'm working on python part :) > RandomForestRegressionModel

[jira] [Updated] (SPARK-19899) FPGrowth input column naming

2017-03-14 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-19899: -- Target Version/s: 2.2.0 > FPGrowth input column naming >

[jira] [Updated] (SPARK-19899) FPGrowth input column naming

2017-03-14 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-19899: -- Shepherd: Joseph K. Bradley > FPGrowth input column naming >

[jira] [Commented] (SPARK-19899) FPGrowth input column naming

2017-03-14 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924362#comment-15924362 ] Joseph K. Bradley commented on SPARK-19899: --- Thanks for bringing this up. I'm pretty convinced

[jira] [Commented] (SPARK-11569) StringIndexer transform fails when column contains nulls

2017-03-14 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924355#comment-15924355 ] Joseph K. Bradley commented on SPARK-11569: --- Linking [SPARK-19852], which can update the Python

[jira] [Assigned] (SPARK-11569) StringIndexer transform fails when column contains nulls

2017-03-14 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley reassigned SPARK-11569: - Assignee: Menglong TAN > StringIndexer transform fails when column contains

[jira] [Resolved] (SPARK-11569) StringIndexer transform fails when column contains nulls

2017-03-14 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-11569. --- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17233

[jira] [Updated] (SPARK-11569) StringIndexer transform fails when column contains nulls

2017-03-14 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-11569: -- Issue Type: Improvement (was: Bug) > StringIndexer transform fails when column

[jira] [Comment Edited] (SPARK-19553) Add GroupedData.countApprox()

2017-03-14 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870780#comment-15870780 ] Nicholas Chammas edited comment on SPARK-19553 at 3/14/17 2:38 PM: --- The

[jira] [Resolved] (SPARK-19940) FPGrowthModel.transform should skip duplicated items

2017-03-14 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-19940. --- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17283

[jira] [Assigned] (SPARK-19940) FPGrowthModel.transform should skip duplicated items

2017-03-14 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley reassigned SPARK-19940: - Assignee: Maciej Szymkiewicz > FPGrowthModel.transform should skip duplicated

[jira] [Assigned] (SPARK-19949) unify bad record handling in CSV and JSON

2017-03-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19949: Assignee: Apache Spark (was: Wenchen Fan) > unify bad record handling in CSV and JSON >

  1   2   >