[jira] [Assigned] (SPARK-22319) SparkSubmit calls getFileStatus before calling loginUserFromKeytab

2017-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22319: Assignee: (was: Apache Spark) > SparkSubmit calls getFileStatus before calling

[jira] [Commented] (SPARK-22319) SparkSubmit calls getFileStatus before calling loginUserFromKeytab

2017-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212161#comment-16212161 ] Apache Spark commented on SPARK-22319: -- User 'sjrand' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22319) SparkSubmit calls getFileStatus before calling loginUserFromKeytab

2017-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22319: Assignee: Apache Spark > SparkSubmit calls getFileStatus before calling

[jira] [Commented] (SPARK-22291) Postgresql UUID[] to Cassandra: Conversion Error

2017-10-19 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212154#comment-16212154 ] Liang-Chi Hsieh commented on SPARK-22291: - Could you send a PR for this? > Postgresql UUID[] to

[jira] [Created] (SPARK-22319) SparkSubmit calls getFileStatus before calling loginUserFromKeytab

2017-10-19 Thread Steven Rand (JIRA)
Steven Rand created SPARK-22319: --- Summary: SparkSubmit calls getFileStatus before calling loginUserFromKeytab Key: SPARK-22319 URL: https://issues.apache.org/jira/browse/SPARK-22319 Project: Spark

[jira] [Commented] (SPARK-22296) CodeGenerator - failed to compile when constructor has scala.collection.mutable.Seq vs. scala.collection.Seq

2017-10-19 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212123#comment-16212123 ] Liang-Chi Hsieh commented on SPARK-22296: - Seems no problem with 2.2? {code} scala> case class

[jira] [Created] (SPARK-22318) spark stream Kafka hang at JavaStreamingContext.start, no spark job create

2017-10-19 Thread iceriver322 (JIRA)
iceriver322 created SPARK-22318: --- Summary: spark stream Kafka hang at JavaStreamingContext.start, no spark job create Key: SPARK-22318 URL: https://issues.apache.org/jira/browse/SPARK-22318 Project:

[jira] [Resolved] (SPARK-22026) data source v2 write path

2017-10-19 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22026. - Resolution: Fixed Fix Version/s: 2.3.0 > data source v2 write path > - >

[jira] [Commented] (SPARK-22312) Spark job stuck with no executor due to bug in Executor Allocation Manager

2017-10-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212071#comment-16212071 ] Saisai Shao commented on SPARK-22312: - I think it is duplicated to SPARK-11334 according to your

[jira] [Created] (SPARK-22317) Spark Thrift job with HTTP ERROR 500

2017-10-19 Thread hoangle (JIRA)
hoangle created SPARK-22317: --- Summary: Spark Thrift job with HTTP ERROR 500 Key: SPARK-22317 URL: https://issues.apache.org/jira/browse/SPARK-22317 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-22209) PySpark does not recognize imports from submodules

2017-10-19 Thread Joel Croteau (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212027#comment-16212027 ] Joel Croteau commented on SPARK-22209: -- Yes [~bryanc], I know I can do that. It's just annoying that

[jira] [Assigned] (SPARK-22268) Fix java style errors

2017-10-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-22268: Assignee: Andrew Ash > Fix java style errors > - > >

[jira] [Resolved] (SPARK-22268) Fix java style errors

2017-10-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-22268. -- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19486

[jira] [Commented] (SPARK-19357) Parallel Model Evaluation for ML Tuning: Scala

2017-10-19 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211923#comment-16211923 ] Joseph K. Bradley commented on SPARK-19357: --- Linking [SPARK-22126], which tracks improving the

[jira] [Commented] (SPARK-22284) Code of class \"org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection\" grows beyond 64 KB

2017-10-19 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211906#comment-16211906 ] Dongjoon Hyun commented on SPARK-22284: --- Ping, [~mgaido]. > Code of class >

[jira] [Commented] (SPARK-21785) Support create table from a file schema

2017-10-19 Thread Jacky Shen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211897#comment-16211897 ] Jacky Shen commented on SPARK-21785: any comment [~dongjoon]? > Support create table from a file

[jira] [Comment Edited] (SPARK-22316) Cannot Select ReducedAggregator Column

2017-10-19 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211849#comment-16211849 ] Russell Spitzer edited comment on SPARK-22316 at 10/19/17 10:21 PM:

[jira] [Commented] (SPARK-22316) Cannot Select ReducedAggregator Column

2017-10-19 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211849#comment-16211849 ] Russell Spitzer commented on SPARK-22316: - Nope I'm allowed to have columns with parens in this

[jira] [Updated] (SPARK-22316) Cannot Select ReducedAggregator Column

2017-10-19 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Russell Spitzer updated SPARK-22316: Description: Given a dataset which has been run through reduceGroups like this {code} case

[jira] [Updated] (SPARK-22316) Cannot Select ReducedAggregator Column

2017-10-19 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Russell Spitzer updated SPARK-22316: Description: Given a dataset which has been run through reduceGroups like this {code} case

[jira] [Commented] (SPARK-22316) Cannot Select ReducedAggregator Column

2017-10-19 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211846#comment-16211846 ] Russell Spitzer commented on SPARK-22316: - You are right, i meant to have createDataset up there.

[jira] [Commented] (SPARK-22209) PySpark does not recognize imports from submodules

2017-10-19 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211772#comment-16211772 ] Bryan Cutler commented on SPARK-22209: -- As a workaround, you could probably do the following {code}

[jira] [Commented] (SPARK-22250) Be less restrictive on type checking

2017-10-19 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211750#comment-16211750 ] Bryan Cutler commented on SPARK-22250: -- [~ferdonline] maybe SPARK-20791 would help you out when

[jira] [Commented] (SPARK-22316) Cannot Select ReducedAggregator Column

2017-10-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211733#comment-16211733 ] Sean Owen commented on SPARK-22316: --- OK, so it's just generally about selecting this column, not

[jira] [Commented] (SPARK-22306) INFER_AND_SAVE overwrites important metadata in Parquet Metastore table

2017-10-19 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211670#comment-16211670 ] Dongjoon Hyun commented on SPARK-22306: --- Thank you, [~budde]! > INFER_AND_SAVE overwrites

[jira] [Commented] (SPARK-22306) INFER_AND_SAVE overwrites important metadata in Parquet Metastore table

2017-10-19 Thread Adam Budde (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211642#comment-16211642 ] Adam Budde commented on SPARK-22306: I'll take a look and potentially start on a patch as soon as I

[jira] [Commented] (SPARK-22306) INFER_AND_SAVE overwrites important metadata in Parquet Metastore table

2017-10-19 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211633#comment-16211633 ] Dongjoon Hyun commented on SPARK-22306: --- Hi, [~budde]. Could you take a look this issue in Spark

[jira] [Updated] (SPARK-22306) INFER_AND_SAVE overwrites important metadata in Parquet Metastore table

2017-10-19 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-22306: -- Description: I noticed some critical changes on my hive tables and realized that they were

[jira] [Commented] (SPARK-21033) fix the potential OOM in UnsafeExternalSorter

2017-10-19 Thread Cosmin Lehene (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211576#comment-16211576 ] Cosmin Lehene commented on SPARK-21033: --- [~srowen] I'm hitting this while running some large jobs.

[jira] [Commented] (SPARK-21033) fix the potential OOM in UnsafeExternalSorter

2017-10-19 Thread Cosmin Lehene (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211574#comment-16211574 ] Cosmin Lehene commented on SPARK-21033: --- [~cloud_fan] Can you update the title and description? It

[jira] [Commented] (SPARK-22308) Support unit tests of spark code using ScalaTest using suites other than FunSuite

2017-10-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211561#comment-16211561 ] Sean Owen commented on SPARK-22308: --- Yeah I wouldn't document it. I disagree that it should be an API

[jira] [Commented] (SPARK-21033) fix the potential OOM in UnsafeExternalSorter

2017-10-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211562#comment-16211562 ] Sean Owen commented on SPARK-21033: --- I'm seeing that same problem consistently in a deployment

[jira] [Commented] (SPARK-22308) Support unit tests of spark code using ScalaTest using suites other than FunSuite

2017-10-19 Thread Nathan Kronenfeld (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211546#comment-16211546 ] Nathan Kronenfeld commented on SPARK-22308: --- > Sure, but aren't you saying that the ship has

[jira] [Comment Edited] (SPARK-22316) Cannot Select ReducedAggregator Column

2017-10-19 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211528#comment-16211528 ] Russell Spitzer edited comment on SPARK-22316 at 10/19/17 6:49 PM: --- I

[jira] [Comment Edited] (SPARK-22316) Cannot Select ReducedAggregator Column

2017-10-19 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211528#comment-16211528 ] Russell Spitzer edited comment on SPARK-22316 at 10/19/17 6:46 PM: --- I

[jira] [Updated] (SPARK-22306) INFER_AND_SAVE overwrites important metadata in Parquet Metastore table

2017-10-19 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-22306: -- Description: I noticed some critical changes on my hive tables and realized that they were

[jira] [Commented] (SPARK-22316) Cannot Select ReducedAggregator Column

2017-10-19 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211528#comment-16211528 ] Russell Spitzer commented on SPARK-22316: - I can imagine many reasons i might want to access a

[jira] [Commented] (SPARK-21033) fix the potential OOM in UnsafeExternalSorter

2017-10-19 Thread Cosmin Lehene (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211526#comment-16211526 ] Cosmin Lehene commented on SPARK-21033: --- I think this may be responsible for other problems, such

[jira] [Commented] (SPARK-22306) INFER_AND_SAVE overwrites important metadata in Parquet Metastore table

2017-10-19 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211518#comment-16211518 ] Dongjoon Hyun commented on SPARK-22306: --- When we release 2.2.1, we had better give a warning about

[jira] [Updated] (SPARK-22306) INFER_AND_SAVE overwrites important metadata in Parquet Metastore table

2017-10-19 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-22306: -- Priority: Critical (was: Major) > INFER_AND_SAVE overwrites important metadata in Parquet

[jira] [Updated] (SPARK-22306) INFER_AND_SAVE overwrites important metadata in Parquet Metastore table

2017-10-19 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-22306: -- Description: I noticed some critical changes on my hive tables and realized that they were

[jira] [Commented] (SPARK-22316) Cannot Select ReducedAggregator Column

2017-10-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211507#comment-16211507 ] Sean Owen commented on SPARK-22316: --- Why are you trying to select it -- why is it a bug? > Cannot

[jira] [Updated] (SPARK-22306) INFER_AND_SAVE overwrites important metadata in Parquet Metastore table

2017-10-19 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-22306: -- Description: I noticed some critical changes on my hive tables and realized that they were

[jira] [Updated] (SPARK-22316) Cannot Select ReducedAggregator Column

2017-10-19 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Russell Spitzer updated SPARK-22316: Summary: Cannot Select ReducedAggregator Column (was: Cannot Select ReducedAggreagtor

[jira] [Created] (SPARK-22316) Cannot Select ReducedAggreagtor Column

2017-10-19 Thread Russell Spitzer (JIRA)
Russell Spitzer created SPARK-22316: --- Summary: Cannot Select ReducedAggreagtor Column Key: SPARK-22316 URL: https://issues.apache.org/jira/browse/SPARK-22316 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-22314) Accessing Hive UDFs defined without 'USING JAR' from Spark

2017-10-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211497#comment-16211497 ] Sean Owen commented on SPARK-22314: --- If it's on the classpath, it's visible to the user. This is just

[jira] [Updated] (SPARK-22306) INFER_AND_SAVE overwrites important metadata in Parquet Metastore table

2017-10-19 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-22306: -- Summary: INFER_AND_SAVE overwrites important metadata in Parquet Metastore table (was:

[jira] [Commented] (SPARK-20393) Strengthen Spark to prevent XSS vulnerabilities

2017-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211450#comment-16211450 ] Apache Spark commented on SPARK-20393: -- User 'ambauma' has created a pull request for this issue:

[jira] [Updated] (SPARK-22314) Accessing Hive UDFs defined without 'USING JAR' from Spark

2017-10-19 Thread Matyas Orhidi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matyas Orhidi updated SPARK-22314: -- Issue Type: Improvement (was: Bug) > Accessing Hive UDFs defined without 'USING JAR' from

[jira] [Created] (SPARK-22315) Check for version match between R package and JVM

2017-10-19 Thread Shivaram Venkataraman (JIRA)
Shivaram Venkataraman created SPARK-22315: - Summary: Check for version match between R package and JVM Key: SPARK-22315 URL: https://issues.apache.org/jira/browse/SPARK-22315 Project: Spark

[jira] [Commented] (SPARK-22306) INFER_AND_SAVE overwrites important metadata in Metastore

2017-10-19 Thread David Malinge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211433#comment-16211433 ] David Malinge commented on SPARK-22306: --- With:

[jira] [Commented] (SPARK-22314) Accessing Hive UDFs defined without 'USING JAR' from Spark

2017-10-19 Thread Matyas Orhidi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211434#comment-16211434 ] Matyas Orhidi commented on SPARK-22314: --- There are use cases when the implementation should be

[jira] [Commented] (SPARK-22314) Accessing Hive UDFs defined without 'USING JAR' from Spark

2017-10-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211422#comment-16211422 ] Sean Owen commented on SPARK-22314: --- What is the problem here? either way you must specify the JAR

[jira] [Updated] (SPARK-22314) Accessing Hive UDFs defined without 'USING JAR' from Spark

2017-10-19 Thread Matyas Orhidi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matyas Orhidi updated SPARK-22314: -- Description: When defining UDF functions in Hive it is possible to load the UDF jar(s) from a

[jira] [Updated] (SPARK-22314) Accessing Hive UDFs defined without 'USING JAR' from Spark

2017-10-19 Thread Matyas Orhidi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matyas Orhidi updated SPARK-22314: -- Description: When defining UDF functions in Hive it is possible to load the UDF jar(s) from a

[jira] [Updated] (SPARK-22314) Accessing Hive UDFs defined without 'USING JAR' from Spark

2017-10-19 Thread Matyas Orhidi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matyas Orhidi updated SPARK-22314: -- Summary: Accessing Hive UDFs defined without 'USING JAR' from Spark (was: Accessing Hive

[jira] [Created] (SPARK-22314) Accessing Hive UDFs from Spark originally defined without 'USING JAR'

2017-10-19 Thread Matyas Orhidi (JIRA)
Matyas Orhidi created SPARK-22314: - Summary: Accessing Hive UDFs from Spark originally defined without 'USING JAR' Key: SPARK-22314 URL: https://issues.apache.org/jira/browse/SPARK-22314 Project:

[jira] [Commented] (SPARK-22306) INFER_AND_SAVE overwrites important metadata in Metastore

2017-10-19 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211375#comment-16211375 ] Dongjoon Hyun commented on SPARK-22306: --- Hi. [~WhoisDavid]. Spark 2.3 seems to be okay due to

[jira] [Commented] (SPARK-22307) NOT condition working incorrectly

2017-10-19 Thread Andrey Yakovenko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211354#comment-16211354 ] Andrey Yakovenko commented on SPARK-22307: -- Id is not null for all records but Parent.Id,

[jira] [Commented] (SPARK-22284) Code of class \"org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection\" grows beyond 64 KB

2017-10-19 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211353#comment-16211353 ] Kazuaki Ishizaki commented on SPARK-22284: -- I found [this

[jira] [Commented] (SPARK-22284) Code of class \"org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection\" grows beyond 64 KB

2017-10-19 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211315#comment-16211315 ] Kazuaki Ishizaki commented on SPARK-22284: -- Thank you for uploading the generated code. This is

[jira] [Commented] (SPARK-22308) Support unit tests of spark code using ScalaTest using suites other than FunSuite

2017-10-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211301#comment-16211301 ] Sean Owen commented on SPARK-22308: --- Sure, but aren't you saying that the ship has sailed and someone

[jira] [Commented] (SPARK-22308) Support unit tests of spark code using ScalaTest using suites other than FunSuite

2017-10-19 Thread Nathan Kronenfeld (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211291#comment-16211291 ] Nathan Kronenfeld commented on SPARK-22308: --- That's a perfect example of what I'm talking

[jira] [Commented] (SPARK-22306) INFER_AND_SAVE overwrites important metadata in Metastore

2017-10-19 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211276#comment-16211276 ] Dongjoon Hyun commented on SPARK-22306: --- Thank you for confirming. Sorry for asking again, could

[jira] [Updated] (SPARK-22306) INFER_AND_SAVE overwrites important metadata in Metastore

2017-10-19 Thread David Malinge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Malinge updated SPARK-22306: -- Environment: Hive 2.3.0 (PostgresQL metastore, stored as Parquet) Spark 2.2.0 was: Hive

[jira] [Commented] (SPARK-22307) NOT condition working incorrectly

2017-10-19 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211239#comment-16211239 ] Marco Gaido commented on SPARK-22307: - Have you checked if the missing records contain null as a

[jira] [Commented] (SPARK-22308) Support unit tests of spark code using ScalaTest using suites other than FunSuite

2017-10-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211228#comment-16211228 ] Sean Owen commented on SPARK-22308: --- There are purpose-built Spark testing frameworks that seem built

[jira] [Commented] (SPARK-22308) Support unit tests of spark code using ScalaTest using suites other than FunSuite

2017-10-19 Thread Nathan Kronenfeld (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211207#comment-16211207 ] Nathan Kronenfeld commented on SPARK-22308: --- I can't disagree more strongly. What you get then

[jira] [Commented] (SPARK-22295) Chi Square selector not recognizing field in Data frame

2017-10-19 Thread Cheburakshu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211200#comment-16211200 ] Cheburakshu commented on SPARK-22295: - But, I had one problem with Chi Square on this data set. I

[jira] [Commented] (SPARK-22295) Chi Square selector not recognizing field in Data frame

2017-10-19 Thread Cheburakshu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211199#comment-16211199 ] Cheburakshu commented on SPARK-22295: - Yes. I realized that and marked my ticket as "Invalid" >

[jira] [Commented] (SPARK-20153) Support Multiple aws credentials in order to access multiple Hive on S3 table in spark application

2017-10-19 Thread Kun Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211183#comment-16211183 ] Kun Liu commented on SPARK-20153: - Steve: AFAIK, "Amazon EMR does not currently support use of the Apache

[jira] [Updated] (SPARK-22284) Code of class \"org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection\" grows beyond 64 KB

2017-10-19 Thread Ben (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben updated SPARK-22284: Attachment: 64KB Error.log Sure, I just added it as an attachment. > Code of class >

[jira] [Comment Edited] (SPARK-22306) INFER_AND_SAVE overwrites important metadata in Metastore

2017-10-19 Thread David Malinge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211129#comment-16211129 ] David Malinge edited comment on SPARK-22306 at 10/19/17 2:32 PM: -

[jira] [Commented] (SPARK-22306) INFER_AND_SAVE overwrites important metadata in Metastore

2017-10-19 Thread David Malinge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211129#comment-16211129 ] David Malinge commented on SPARK-22306: --- [~dongjoon]

[jira] [Commented] (SPARK-22229) SPIP: RDMA Accelerated Shuffle Engine

2017-10-19 Thread Yuval Degani (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211038#comment-16211038 ] Yuval Degani commented on SPARK-9: -- [~rvesse], thanks for taking the time to review. Regarding

[jira] [Commented] (SPARK-22254) clean up the implementation of `growToSize` in CompactBuffer

2017-10-19 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210964#comment-16210964 ] Kazuaki Ishizaki commented on SPARK-22254: -- If noone has started working for this, I will work

[jira] [Commented] (SPARK-22284) Code of class \"org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection\" grows beyond 64 KB

2017-10-19 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210943#comment-16210943 ] Kazuaki Ishizaki commented on SPARK-22284: -- Can you attach the generated code? It may help which

[jira] [Comment Edited] (SPARK-20617) pyspark.sql filtering fails when using ~isin when there are nulls in column

2017-10-19 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210898#comment-16210898 ] Marco Gaido edited comment on SPARK-20617 at 10/19/17 11:43 AM: This is

[jira] [Comment Edited] (SPARK-20617) pyspark.sql filtering fails when using ~isin when there are nulls in column

2017-10-19 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210898#comment-16210898 ] Marco Gaido edited comment on SPARK-20617 at 10/19/17 11:43 AM: This is

[jira] [Resolved] (SPARK-20617) pyspark.sql filtering fails when using ~isin when there are nulls in column

2017-10-19 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido resolved SPARK-20617. - Resolution: Not A Bug > pyspark.sql filtering fails when using ~isin when there are nulls in

[jira] [Commented] (SPARK-20617) pyspark.sql filtering fails when using ~isin when there are nulls in column

2017-10-19 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210898#comment-16210898 ] Marco Gaido commented on SPARK-20617: - This is not a bug. This is the right and expected behavior

[jira] [Commented] (SPARK-6685) Use DSYRK to compute AtA in ALS

2017-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210888#comment-16210888 ] Apache Spark commented on SPARK-6685: - User 'mpjlu' has created a pull request for this issue:

[jira] [Updated] (SPARK-22309) Remove unused param in `LDAModel.getTopicDistributionMethod`

2017-10-19 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-22309: - Description: 1, Param {{sc: SparkContext}} in {{LocalLDAModel.getTopicDistributionMethod}} is

[jira] [Updated] (SPARK-22309) Remove unused param in `LDAModel.getTopicDistributionMethod`

2017-10-19 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-22309: - Summary: Remove unused param in `LDAModel.getTopicDistributionMethod` (was: Remove unused param

[jira] [Commented] (SPARK-22309) Remove unused param in `LDAModel.getTopicDistributionMethod` & destory `nodeToFeaturesBc` in RandomForest

2017-10-19 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210838#comment-16210838 ] zhengruifeng commented on SPARK-22309: -- OK, I will separate them > Remove unused param in

[jira] [Assigned] (SPARK-22313) Mark/print deprecation warnings as DeprecationWarning for deprecated APIs

2017-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22313: Assignee: Apache Spark > Mark/print deprecation warnings as DeprecationWarning for

[jira] [Assigned] (SPARK-22313) Mark/print deprecation warnings as DeprecationWarning for deprecated APIs

2017-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22313: Assignee: (was: Apache Spark) > Mark/print deprecation warnings as DeprecationWarning

[jira] [Commented] (SPARK-22313) Mark/print deprecation warnings as DeprecationWarning for deprecated APIs

2017-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210759#comment-16210759 ] Apache Spark commented on SPARK-22313: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Created] (SPARK-22313) Mark/print deprecation warnings as DeprecationWarning for deprecated APIs

2017-10-19 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-22313: Summary: Mark/print deprecation warnings as DeprecationWarning for deprecated APIs Key: SPARK-22313 URL: https://issues.apache.org/jira/browse/SPARK-22313 Project:

[jira] [Assigned] (SPARK-14371) OnlineLDAOptimizer should not collect stats for each doc in mini-batch to driver

2017-10-19 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley reassigned SPARK-14371: - Assignee: Valeriy Avanesov > OnlineLDAOptimizer should not collect stats for

[jira] [Resolved] (SPARK-14371) OnlineLDAOptimizer should not collect stats for each doc in mini-batch to driver

2017-10-19 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-14371. --- Resolution: Fixed Fix Version/s: 2.3.0 > OnlineLDAOptimizer should not

[jira] [Assigned] (SPARK-22188) Add defense against Cross-Site Scripting, MIME-sniffing and MitM attack

2017-10-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-22188: - Assignee: Krishna Pandey > Add defense against Cross-Site Scripting, MIME-sniffing and MitM

[jira] [Resolved] (SPARK-22188) Add defense against Cross-Site Scripting, MIME-sniffing and MitM attack

2017-10-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22188. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19419

[jira] [Commented] (SPARK-22308) Support unit tests of spark code using ScalaTest using suites other than FunSuite

2017-10-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210692#comment-16210692 ] Sean Owen commented on SPARK-22308: --- These are internal test classes and not any kind of API, so I

[jira] [Resolved] (SPARK-22288) Tricky interaction between closure-serialization and inheritance results in confusing failure

2017-10-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22288. --- Resolution: Not A Problem OK, fine to note here, but I think this is just a Java issue. You're right

[jira] [Updated] (SPARK-22309) Remove unused param in `LDAModel.getTopicDistributionMethod` & destory `nodeToFeaturesBc` in RandomForest

2017-10-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-22309: -- Priority: Trivial (was: Minor) Let's not mix unrelated issues please > Remove unused param in

[jira] [Resolved] (SPARK-22311) stage api modify the description format, add version api, modify the duration real-time calculation

2017-10-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22311. --- Resolution: Won't Fix This is too trivial for a JIRA, and one of the changes is wrong. See comments

[jira] [Resolved] (SPARK-22283) withColumn should replace multiple instances with a single one

2017-10-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22283. --- Resolution: Not A Problem > withColumn should replace multiple instances with a single one >

[jira] [Resolved] (SPARK-22290) Starting second context in same JVM fails to get new Hive delegation token

2017-10-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22290. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19509

[jira] [Assigned] (SPARK-22290) Starting second context in same JVM fails to get new Hive delegation token

2017-10-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22290: --- Assignee: Marcelo Vanzin > Starting second context in same JVM fails to get new Hive

  1   2   >