[jira] [Commented] (SPARK-19288) Failure (at test_sparkSQL.R#1300): date functions on a DataFrame in R/run-tests.sh

2017-03-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929438#comment-15929438 ] Hyukjin Kwon commented on SPARK-19288: -- FWIW, for me it has been fine. Mac OS 10.12.3 & KTS & R

[jira] [Commented] (SPARK-19288) Failure (at test_sparkSQL.R#1300): date functions on a DataFrame in R/run-tests.sh

2017-03-16 Thread Miao Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929433#comment-15929433 ] Miao Wang commented on SPARK-19288: --- I think it only happens at local build. I had another similar

[jira] [Comment Edited] (SPARK-19827) spark.ml R API for PIC

2017-03-16 Thread Miao Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929429#comment-15929429 ] Miao Wang edited comment on SPARK-19827 at 3/17/17 5:15 AM: Please hold on.

[jira] [Commented] (SPARK-19827) spark.ml R API for PIC

2017-03-16 Thread Miao Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929429#comment-15929429 ] Miao Wang commented on SPARK-19827: --- Please hold on. We need to add wrapper to ML instead of MLLIB. The

[jira] [Commented] (SPARK-19990) Flaky test: org.apache.spark.sql.hive.execution.HiveCatalogedDDLSuite: create temporary view using

2017-03-16 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929422#comment-15929422 ] Kay Ousterhout commented on SPARK-19990: Thanks [~windpiger]! > Flaky test:

[jira] [Comment Edited] (SPARK-19990) Flaky test: org.apache.spark.sql.hive.execution.HiveCatalogedDDLSuite: create temporary view using

2017-03-16 Thread Song Jun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929409#comment-15929409 ] Song Jun edited comment on SPARK-19990 at 3/17/17 4:36 AM: --- the root cause is

[jira] [Comment Edited] (SPARK-19990) Flaky test: org.apache.spark.sql.hive.execution.HiveCatalogedDDLSuite: create temporary view using

2017-03-16 Thread Song Jun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929409#comment-15929409 ] Song Jun edited comment on SPARK-19990 at 3/17/17 4:35 AM: --- the root cause is

[jira] [Commented] (SPARK-19990) Flaky test: org.apache.spark.sql.hive.execution.HiveCatalogedDDLSuite: create temporary view using

2017-03-16 Thread Song Jun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929409#comment-15929409 ] Song Jun commented on SPARK-19990: -- the root cause is [the csvfile path in this test

[jira] [Comment Edited] (SPARK-19984) ERROR codegen.CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java'

2017-03-16 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929399#comment-15929399 ] Kazuaki Ishizaki edited comment on SPARK-19984 at 3/17/17 4:23 AM: ---

[jira] [Commented] (SPARK-19984) ERROR codegen.CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java'

2017-03-16 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929399#comment-15929399 ] Kazuaki Ishizaki commented on SPARK-19984: -- This problem occurs since Spark generates

[jira] [Created] (SPARK-19991) FileSegmentManagedBuffer performance improvement.

2017-03-16 Thread Guoqiang Li (JIRA)
Guoqiang Li created SPARK-19991: --- Summary: FileSegmentManagedBuffer performance improvement. Key: SPARK-19991 URL: https://issues.apache.org/jira/browse/SPARK-19991 Project: Spark Issue Type:

[jira] [Updated] (SPARK-19991) FileSegmentManagedBuffer performance improvement.

2017-03-16 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-19991: Description: When we do not set the value of the configuration items

[jira] [Updated] (SPARK-19736) refreshByPath should clear all cached plans with the specified path

2017-03-16 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-19736: Fix Version/s: 2.1.1 > refreshByPath should clear all cached plans with the specified path >

[jira] [Updated] (SPARK-19093) Cached tables are not used in SubqueryExpression

2017-03-16 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-19093: Fix Version/s: 2.1.1 > Cached tables are not used in SubqueryExpression >

[jira] [Updated] (SPARK-18549) Failed to Uncache a View that References a Dropped Table.

2017-03-16 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-18549: Fix Version/s: 2.1.1 > Failed to Uncache a View that References a Dropped Table. >

[jira] [Updated] (SPARK-19765) UNCACHE TABLE should also un-cache all cached plans that refer to this table

2017-03-16 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-19765: Fix Version/s: 2.1.1 > UNCACHE TABLE should also un-cache all cached plans that refer to this

[jira] [Commented] (SPARK-19975) Add map_keys and map_values functions to Python

2017-03-16 Thread Yong Tang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929342#comment-15929342 ] Yong Tang commented on SPARK-19975: --- Created a PR for that: https://github.com/apache/spark/pull/17328

[jira] [Updated] (SPARK-19990) Flaky test: org.apache.spark.sql.hive.execution.HiveCatalogedDDLSuite: create temporary view using

2017-03-16 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kay Ousterhout updated SPARK-19990: --- Description: This test seems to be failing consistently on all of the maven builds:

[jira] [Commented] (SPARK-19988) Flaky Test: OrcSourceSuite SPARK-19459/SPARK-18220: read char/varchar column written by Hive

2017-03-16 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929334#comment-15929334 ] Kay Ousterhout commented on SPARK-19988: With some help from [~joshrosen] I spent some time

[jira] [Created] (SPARK-19990) Flaky test: org.apache.spark.sql.hive.execution.HiveCatalogedDDLSuite: create temporary view using

2017-03-16 Thread Kay Ousterhout (JIRA)
Kay Ousterhout created SPARK-19990: -- Summary: Flaky test: org.apache.spark.sql.hive.execution.HiveCatalogedDDLSuite: create temporary view using Key: SPARK-19990 URL:

[jira] [Resolved] (SPARK-19987) Pass all filters into FileIndex

2017-03-16 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-19987. - Resolution: Fixed Fix Version/s: 2.2.0 > Pass all filters into FileIndex >

[jira] [Commented] (SPARK-19982) JavaDatasetSuite.testJavaBeanEncoder sometimes fails with "Unable to generate an encoder for inner class"

2017-03-16 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929296#comment-15929296 ] Wenchen Fan commented on SPARK-19982: - yea makes sense, the test harness should hold the {{this}}

[jira] [Commented] (SPARK-19982) JavaDatasetSuite.testJavaBeanEncoder sometimes fails with "Unable to generate an encoder for inner class"

2017-03-16 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929275#comment-15929275 ] Michael Armbrust commented on SPARK-19982: -- I'm not sure if changing weak to strong references

[jira] [Commented] (SPARK-18789) Save Data frame with Null column-- exception

2017-03-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929272#comment-15929272 ] Hyukjin Kwon commented on SPARK-18789: -- Do you mind if I ask a simple code for this? Pseudocode is

[jira] [Commented] (SPARK-19969) Doc and examples for Imputer

2017-03-16 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929271#comment-15929271 ] yuhao yang commented on SPARK-19969: Looks like jira stops auto binding with PR.

[jira] [Commented] (SPARK-18789) Save Data frame with Null column-- exception

2017-03-16 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929259#comment-15929259 ] Harish commented on SPARK-18789: When you create the DF (dynamic) withough knowing the type of the column

[jira] [Updated] (SPARK-19964) Flaky test: SparkSubmitSuite fails due to Timeout

2017-03-16 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kay Ousterhout updated SPARK-19964: --- Summary: Flaky test: SparkSubmitSuite fails due to Timeout (was: SparkSubmitSuite fails due

[jira] [Comment Edited] (SPARK-19964) Flaky test: SparkSubmitSuite fails due to Timeout

2017-03-16 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929257#comment-15929257 ] Kay Ousterhout edited comment on SPARK-19964 at 3/17/17 12:54 AM: --

[jira] [Commented] (SPARK-19964) SparkSubmitSuite fails due to Timeout

2017-03-16 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929257#comment-15929257 ] Kay Ousterhout commented on SPARK-19964: [~srowen] it looks like this is failing periodically in

[jira] [Commented] (SPARK-19982) JavaDatasetSuite.testJavaBeanEncoder sometimes fails with "Unable to generate an encoder for inner class"

2017-03-16 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929250#comment-15929250 ] Wenchen Fan commented on SPARK-19982: - I think this is caused by weak references, a GC may happen

[jira] [Resolved] (SPARK-19635) Feature parity for Chi-square hypothesis testing in MLlib

2017-03-16 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-19635. --- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17110

[jira] [Updated] (SPARK-19988) Flaky Test: OrcSourceSuite SPARK-19459/SPARK-18220: read char/varchar column written by Hive

2017-03-16 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kay Ousterhout updated SPARK-19988: --- Component/s: SQL > Flaky Test: OrcSourceSuite SPARK-19459/SPARK-18220: read char/varchar

[jira] [Created] (SPARK-19989) Flaky Test: org.apache.spark.sql.kafka010.KafkaSourceStressSuite

2017-03-16 Thread Kay Ousterhout (JIRA)
Kay Ousterhout created SPARK-19989: -- Summary: Flaky Test: org.apache.spark.sql.kafka010.KafkaSourceStressSuite Key: SPARK-19989 URL: https://issues.apache.org/jira/browse/SPARK-19989 Project: Spark

[jira] [Commented] (SPARK-19988) Flaky Test: OrcSourceSuite SPARK-19459/SPARK-18220: read char/varchar column written by Hive

2017-03-16 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929107#comment-15929107 ] Herman van Hovell commented on SPARK-19988: --- It is probably some other test changing the

[jira] [Commented] (SPARK-18789) Save Data frame with Null column-- exception

2017-03-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929084#comment-15929084 ] Hyukjin Kwon commented on SPARK-18789: -- It seems it goes failed in schema inference. {code} >>>

[jira] [Updated] (SPARK-19988) Flaky Test: OrcSourceSuite SPARK-19459/SPARK-18220: read char/varchar column written by Hive

2017-03-16 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-19988: - Attachment: trimmed-unit-test.log Attaching a trimmed version of the unit-test.log file, though

[jira] [Created] (SPARK-19988) Flaky Test: OrcSourceSuite SPARK-19459/SPARK-18220: read char/varchar column written by Hive

2017-03-16 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-19988: Summary: Flaky Test: OrcSourceSuite SPARK-19459/SPARK-18220: read char/varchar column written by Hive Key: SPARK-19988 URL: https://issues.apache.org/jira/browse/SPARK-19988

[jira] [Updated] (SPARK-19987) Pass all filters into FileIndex

2017-03-16 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-19987: Description: This is a tiny teeny refactoring to pass data filters also to the FileIndex, so

[jira] [Updated] (SPARK-19982) JavaDatasetSuite.testJavaBeanEncoder sometimes fails with "Unable to generate an encoder for inner class"

2017-03-16 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-19982: - Labels: flaky-test (was: ) > JavaDatasetSuite.testJavaBeanEncoder sometimes fails with "Unable

[jira] [Created] (SPARK-19987) Pass all filters into FileIndex

2017-03-16 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-19987: --- Summary: Pass all filters into FileIndex Key: SPARK-19987 URL: https://issues.apache.org/jira/browse/SPARK-19987 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-12664) Expose raw prediction scores in MultilayerPerceptronClassificationModel

2017-03-16 Thread Drew Robb (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928928#comment-15928928 ] Drew Robb commented on SPARK-12664: --- This feature is also very important to me. I'm considering working

[jira] [Commented] (SPARK-18789) Save Data frame with Null column-- exception

2017-03-16 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928926#comment-15928926 ] Harish commented on SPARK-18789: In your example you are defining the schema first and then loading the

[jira] [Updated] (SPARK-19985) Some ML Models error when copy or do not set parent

2017-03-16 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-19985: - Description: Some ML Models fail when copied due to not having a default constructor and

[jira] [Created] (SPARK-19986) Make pyspark.streaming.tests.CheckpointTests more stable

2017-03-16 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-19986: Summary: Make pyspark.streaming.tests.CheckpointTests more stable Key: SPARK-19986 URL: https://issues.apache.org/jira/browse/SPARK-19986 Project: Spark

[jira] [Commented] (SPARK-19985) Some ML Models error when copy or do not set parent

2017-03-16 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928924#comment-15928924 ] Bryan Cutler commented on SPARK-19985: -- I'll fix this > Some ML Models error when copy or do not

[jira] [Created] (SPARK-19985) Some ML Models error when copy or do not set parent

2017-03-16 Thread Bryan Cutler (JIRA)
Bryan Cutler created SPARK-19985: Summary: Some ML Models error when copy or do not set parent Key: SPARK-19985 URL: https://issues.apache.org/jira/browse/SPARK-19985 Project: Spark Issue

[jira] [Created] (SPARK-19984) ERROR codegen.CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java'

2017-03-16 Thread Andrey Yakovenko (JIRA)
Andrey Yakovenko created SPARK-19984: Summary: ERROR codegen.CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java' Key: SPARK-19984 URL:

[jira] [Commented] (SPARK-19969) Doc and examples for Imputer

2017-03-16 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928854#comment-15928854 ] Nick Pentreath commented on SPARK-19969: Ok - I can help on it but probably only some time next

[jira] [Commented] (SPARK-19899) FPGrowth input column naming

2017-03-16 Thread Maciej Szymkiewicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928827#comment-15928827 ] Maciej Szymkiewicz commented on SPARK-19899: [~mlnick] For some reason SparkQA recognized the

[jira] [Commented] (SPARK-19713) saveAsTable

2017-03-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928802#comment-15928802 ] Hyukjin Kwon commented on SPARK-19713: -- > please suggest what you think the title should be

[jira] [Commented] (SPARK-18789) Save Data frame with Null column-- exception

2017-03-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928780#comment-15928780 ] Hyukjin Kwon commented on SPARK-18789: -- Hm, do you mind if I ask a reproducer? {code} from

[jira] [Comment Edited] (SPARK-19713) saveAsTable

2017-03-16 Thread Eric Maynard (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928778#comment-15928778 ] Eric Maynard edited comment on SPARK-19713 at 3/16/17 8:08 PM: --- Not really

[jira] [Commented] (SPARK-19713) saveAsTable

2017-03-16 Thread Eric Maynard (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928778#comment-15928778 ] Eric Maynard commented on SPARK-19713: -- Not really relevant here, but to address: >2. Hive cannot

[jira] [Assigned] (SPARK-19721) Good error message for version mismatch in log files

2017-03-16 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-19721: Assignee: Liwei Lin > Good error message for version mismatch in log files >

[jira] [Updated] (SPARK-19721) Good error message for version mismatch in log files

2017-03-16 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19721: - Fix Version/s: 2.2.0 > Good error message for version mismatch in log files >

[jira] [Commented] (SPARK-19713) saveAsTable

2017-03-16 Thread Balaram R Gadiraju (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928737#comment-15928737 ] Balaram R Gadiraju commented on SPARK-19713: @Hyukjin Kwon : please suggest what you think

[jira] [Commented] (SPARK-18789) Save Data frame with Null column-- exception

2017-03-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928738#comment-15928738 ] Hyukjin Kwon commented on SPARK-18789: -- Doh, I am sorry. Let me try to test again and will open. I

[jira] [Commented] (SPARK-19713) saveAsTable

2017-03-16 Thread Balaram R Gadiraju (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928734#comment-15928734 ] Balaram R Gadiraju commented on SPARK-19713: The issue is not only in spark, because when the

[jira] [Commented] (SPARK-19982) JavaDatasetSuite.testJavaBeanEncoder sometimes fails with "Unable to generate an encoder for inner class"

2017-03-16 Thread Jose Soltren (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928660#comment-15928660 ] Jose Soltren commented on SPARK-19982: -- [~cloud_fan] added this test as some work related to

[jira] [Commented] (SPARK-19965) DataFrame batch reader may fail to infer partitions when reading FileStreamSink's output

2017-03-16 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928657#comment-15928657 ] Shixiong Zhu commented on SPARK-19965: -- [~lwlin] I think we can just ignore “_spark_metadata” in

[jira] [Created] (SPARK-19983) Getting ValidationFailureSemanticException on 'INSERT OVEWRITE'

2017-03-16 Thread Rajkumar (JIRA)
Rajkumar created SPARK-19983: Summary: Getting ValidationFailureSemanticException on 'INSERT OVEWRITE' Key: SPARK-19983 URL: https://issues.apache.org/jira/browse/SPARK-19983 Project: Spark

[jira] [Comment Edited] (SPARK-12261) pyspark crash for large dataset

2017-03-16 Thread Tomas Pranckevicius (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927746#comment-15927746 ] Tomas Pranckevicius edited comment on SPARK-12261 at 3/16/17 6:39 PM:

[jira] [Comment Edited] (SPARK-12261) pyspark crash for large dataset

2017-03-16 Thread Tomas Pranckevicius (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927746#comment-15927746 ] Tomas Pranckevicius edited comment on SPARK-12261 at 3/16/17 6:40 PM:

[jira] [Assigned] (SPARK-19945) Add test suite for SessionCatalog with HiveExternalCatalog

2017-03-16 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-19945: --- Assignee: Song Jun > Add test suite for SessionCatalog with HiveExternalCatalog >

[jira] [Resolved] (SPARK-19945) Add test suite for SessionCatalog with HiveExternalCatalog

2017-03-16 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-19945. - Resolution: Fixed Fix Version/s: 2.2.0 > Add test suite for SessionCatalog with

[jira] [Created] (SPARK-19982) JavaDatasetSuite.testJavaBeanEncoder sometimes fails with "Unable to generate an encoder for inner class"

2017-03-16 Thread Jose Soltren (JIRA)
Jose Soltren created SPARK-19982: Summary: JavaDatasetSuite.testJavaBeanEncoder sometimes fails with "Unable to generate an encoder for inner class" Key: SPARK-19982 URL:

[jira] [Comment Edited] (SPARK-19969) Doc and examples for Imputer

2017-03-16 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928549#comment-15928549 ] yuhao yang edited comment on SPARK-19969 at 3/16/17 6:10 PM: - Not really. But

[jira] [Commented] (SPARK-19969) Doc and examples for Imputer

2017-03-16 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928549#comment-15928549 ] yuhao yang commented on SPARK-19969: Not really. But I can start on it now. > Doc and examples for

[jira] [Commented] (SPARK-19979) [MLLIB] Multiple Estimators/Pipelines In CrossValidator

2017-03-16 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928545#comment-15928545 ] Nick Pentreath commented on SPARK-19979: I wonder if this fits in as a sort of sub-task of

[jira] [Commented] (SPARK-19969) Doc and examples for Imputer

2017-03-16 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928537#comment-15928537 ] Nick Pentreath commented on SPARK-19969: No haven't done the doc or examples - I seem to recall

[jira] [Comment Edited] (SPARK-12261) pyspark crash for large dataset

2017-03-16 Thread Tomas Pranckevicius (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927746#comment-15927746 ] Tomas Pranckevicius edited comment on SPARK-12261 at 3/16/17 5:40 PM:

[jira] [Comment Edited] (SPARK-12261) pyspark crash for large dataset

2017-03-16 Thread Tomas Pranckevicius (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927746#comment-15927746 ] Tomas Pranckevicius edited comment on SPARK-12261 at 3/16/17 5:40 PM:

[jira] [Comment Edited] (SPARK-12261) pyspark crash for large dataset

2017-03-16 Thread Tomas Pranckevicius (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927746#comment-15927746 ] Tomas Pranckevicius edited comment on SPARK-12261 at 3/16/17 5:39 PM:

[jira] [Updated] (SPARK-19329) after alter a datasource table's location to a not exist location and then insert data throw Exception

2017-03-16 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-19329: Fix Version/s: 2.1.1 > after alter a datasource table's location to a not exist location and then >

[jira] [Commented] (SPARK-19969) Doc and examples for Imputer

2017-03-16 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928443#comment-15928443 ] yuhao yang commented on SPARK-19969: Thanks for the great help with Imputer, [~mlnick] Have you

[jira] [Commented] (SPARK-14438) Cross-publish Breeze for Scala 2.12

2017-03-16 Thread Kirill chebba Chebunin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928356#comment-15928356 ] Kirill chebba Chebunin commented on SPARK-14438: 2.12 support was added for version 0.13

[jira] [Created] (SPARK-19981) Sort-Merge join inserts shuffles when joining dataframes with aliased columns

2017-03-16 Thread Allen George (JIRA)
Allen George created SPARK-19981: Summary: Sort-Merge join inserts shuffles when joining dataframes with aliased columns Key: SPARK-19981 URL: https://issues.apache.org/jira/browse/SPARK-19981

[jira] [Updated] (SPARK-19980) Basic Dataset transformation on POJOs does not preserves nulls.

2017-03-16 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Lemay updated SPARK-19980: - Description: Applying an identity map transformation on a statically typed Dataset with a POJO

[jira] [Updated] (SPARK-19980) Basic Dataset transformation on POJOs does not preserves nulls.

2017-03-16 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Lemay updated SPARK-19980: - Description: Applying an identity map transformation on a statically typed Dataset with a POJO

[jira] [Created] (SPARK-19980) Basic Dataset transformation on POJOs does not preserves nulls.

2017-03-16 Thread Michel Lemay (JIRA)
Michel Lemay created SPARK-19980: Summary: Basic Dataset transformation on POJOs does not preserves nulls. Key: SPARK-19980 URL: https://issues.apache.org/jira/browse/SPARK-19980 Project: Spark

[jira] [Commented] (SPARK-18789) Save Data frame with Null column-- exception

2017-03-16 Thread Eugen Prokhorenko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928213#comment-15928213 ] Eugen Prokhorenko commented on SPARK-18789: --- Just wanted to mention that initial problem

[jira] [Comment Edited] (SPARK-19977) Scheduler Delay (in UI Advanced Metrics) for a task gradually increases from 5 ms to 30 seconds in Spark Streaming application

2017-03-16 Thread Ray Qiu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928196#comment-15928196 ] Ray Qiu edited comment on SPARK-19977 at 3/16/17 2:45 PM: -- One thing to add is

[jira] [Commented] (SPARK-15040) PySpark impl for ml.feature.Imputer

2017-03-16 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928180#comment-15928180 ] Nick Pentreath commented on SPARK-15040: Sorry, I did not see your comment - I opened a

[jira] [Comment Edited] (SPARK-19977) Scheduler Delay (in UI Advanced Metrics) for a task gradually increases from 5 ms to 30 seconds in Spark Streaming application

2017-03-16 Thread Ray Qiu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928196#comment-15928196 ] Ray Qiu edited comment on SPARK-19977 at 3/16/17 2:45 PM: -- One thing to add is

[jira] [Commented] (SPARK-19977) Scheduler Delay (in UI Advanced Metrics) for a task gradually increases from 5 ms to 30 seconds in Spark Streaming application

2017-03-16 Thread Ray Qiu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928196#comment-15928196 ] Ray Qiu commented on SPARK-19977: - One thing to add is that the same application will not have this issue

[jira] [Commented] (SPARK-19899) FPGrowth input column naming

2017-03-16 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928193#comment-15928193 ] Nick Pentreath commented on SPARK-19899: +1 on {{itemsCol}} - feel free to send a PR :) >

[jira] [Commented] (SPARK-19979) [MLLIB] Multiple Estimators/Pipelines In CrossValidator

2017-03-16 Thread David Leifker (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928195#comment-15928195 ] David Leifker commented on SPARK-19979: --- My apologies, I got a little ahead of this with a proposed

[jira] [Created] (SPARK-19979) [MLLIB] Multiple Estimators/Pipelines In CrossValidator

2017-03-16 Thread David Leifker (JIRA)
David Leifker created SPARK-19979: - Summary: [MLLIB] Multiple Estimators/Pipelines In CrossValidator Key: SPARK-19979 URL: https://issues.apache.org/jira/browse/SPARK-19979 Project: Spark

[jira] [Comment Edited] (SPARK-12261) pyspark crash for large dataset

2017-03-16 Thread Shea Parkes (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928179#comment-15928179 ] Shea Parkes edited comment on SPARK-12261 at 3/16/17 2:38 PM: -- I simply

[jira] [Comment Edited] (SPARK-12261) pyspark crash for large dataset

2017-03-16 Thread Shea Parkes (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928179#comment-15928179 ] Shea Parkes edited comment on SPARK-12261 at 3/16/17 2:38 PM: -- I simply

[jira] [Commented] (SPARK-12261) pyspark crash for large dataset

2017-03-16 Thread Shea Parkes (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928179#comment-15928179 ] Shea Parkes commented on SPARK-12261: - I simply added the following to the end: for _ in iterator:

[jira] [Commented] (SPARK-10764) Add optional caching to Pipelines

2017-03-16 Thread Sachin Tyagi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928169#comment-15928169 ] Sachin Tyagi commented on SPARK-10764: -- Hi, I want to take a stab at it. Here's how I am trying to

[jira] [Commented] (SPARK-19977) Scheduler Delay (in UI Advanced Metrics) for a task gradually increases from 5 ms to 30 seconds in Spark Streaming application

2017-03-16 Thread Ray Qiu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928164#comment-15928164 ] Ray Qiu commented on SPARK-19977: - Not really. Many of the batches are empty RDDs, and the scheduler

[jira] [Comment Edited] (SPARK-19977) Scheduler Delay (in UI Advanced Metrics) for a task gradually increases from 5 ms to 30 seconds in Spark Streaming application

2017-03-16 Thread Ray Qiu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928164#comment-15928164 ] Ray Qiu edited comment on SPARK-19977 at 3/16/17 2:28 PM: -- Not really. Many of

[jira] [Resolved] (SPARK-19946) DebugFilesystem.assertNoOpenStreams should report the open streams to help debugging

2017-03-16 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-19946. --- Resolution: Fixed Assignee: Bogdan Raducanu Fix Version/s: 2.2.0 >

[jira] [Commented] (SPARK-19962) add DictVectorizor for DataFrame

2017-03-16 Thread yu peng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928123#comment-15928123 ] yu peng commented on SPARK-19962: - yeah, exactly.. i would love to use FeatureHasher when i have a lot of

[jira] [Updated] (SPARK-19932) Disallow a case that might cause OOM for steaming deduplication

2017-03-16 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liwei Lin updated SPARK-19932: -- Summary: Disallow a case that might cause OOM for steaming deduplication (was: Disallow a case that

[jira] [Resolved] (SPARK-6678) select count(DISTINCT C_UID) from parquetdir may be can optimize

2017-03-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-6678. - Resolution: Not A Problem I am resolving this as this code path has been radically changed. I

[jira] [Updated] (SPARK-19932) Disallow a case that might case OOM for steaming deduplication

2017-03-16 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liwei Lin updated SPARK-19932: -- Summary: Disallow a case that might case OOM for steaming deduplication (was: Also save event time

[jira] [Commented] (SPARK-18579) spark-csv strips whitespace (pyspark)

2017-03-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928065#comment-15928065 ] Hyukjin Kwon commented on SPARK-18579: -- I submitted a PR for this

  1   2   >