[jira] [Resolved] (SPARK-21549) Spark fails to complete job correctly in case of OutputFormat which do not write into hdfs

[ https://issues.apache.org/jira/browse/SPARK-21549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-21549. - Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 Issue

[jira] [Created] (SPARK-22221) Add User Documentation for Working with Arrow in Spark

Bryan Cutler created SPARK-1: Summary: Add User Documentation for Working with Arrow in Spark Key: SPARK-1 URL: https://issues.apache.org/jira/browse/SPARK-1 Project: Spark Issue

[jira] [Commented] (SPARK-22136) Implement stream-stream outer joins in append mode

[ https://issues.apache.org/jira/browse/SPARK-22136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195221#comment-16195221 ] Apache Spark commented on SPARK-22136: -- User 'joseph-torres' has created a pull request for this

[jira] [Updated] (SPARK-14387) Enable Hive-1.x ORC compatibility with spark.sql.hive.convertMetastoreOrc

[ https://issues.apache.org/jira/browse/SPARK-14387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-14387: -- Affects Version/s: 2.1.1 2.2.0 > Enable Hive-1.x ORC compatibility with

[jira] [Resolved] (SPARK-21947) monotonically_increasing_id doesn't work in Structured Streaming

[ https://issues.apache.org/jira/browse/SPARK-21947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-21947. -- Resolution: Fixed Assignee: Liang-Chi Hsieh Fix Version/s: 2.3.0 >

[jira] [Resolved] (SPARK-22214) Refactor the list hive partitions code

[ https://issues.apache.org/jira/browse/SPARK-22214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22214. - Resolution: Fixed Assignee: Jiang Xingbo Fix Version/s: 2.3.0 > Refactor the list hive

[jira] [Commented] (SPARK-18755) Add Randomized Grid Search to Spark ML

[ https://issues.apache.org/jira/browse/SPARK-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195145#comment-16195145 ] Ilya Matiach commented on SPARK-18755: -- This is a very interesting issue. I am thinking of

[jira] [Updated] (SPARK-22181) ReplaceExceptWithNotFilter if one or both of the datasets are fully derived out of Filters from a same parent

[ https://issues.apache.org/jira/browse/SPARK-22181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sathiya Kumar updated SPARK-22181: -- Description: While applying Except operator between two datasets, if one or both of the

[jira] [Commented] (SPARK-22181) ReplaceExceptWithNotFilter if one or both of the datasets are fully derived out of Filters from a same parent

[ https://issues.apache.org/jira/browse/SPARK-22181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195063#comment-16195063 ] Apache Spark commented on SPARK-22181: -- User 'sathiyapk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22181) ReplaceExceptWithNotFilter if one or both of the datasets are fully derived out of Filters from a same parent

[ https://issues.apache.org/jira/browse/SPARK-22181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22181: Assignee: (was: Apache Spark) > ReplaceExceptWithNotFilter if one or both of the

[jira] [Assigned] (SPARK-22181) ReplaceExceptWithNotFilter if one or both of the datasets are fully derived out of Filters from a same parent

[ https://issues.apache.org/jira/browse/SPARK-22181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22181: Assignee: Apache Spark > ReplaceExceptWithNotFilter if one or both of the datasets are

[jira] [Commented] (SPARK-22218) spark shuffle services fails to update secret on application re-attempts

[ https://issues.apache.org/jira/browse/SPARK-22218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195037#comment-16195037 ] Apache Spark commented on SPARK-22218: -- User 'tgravescs' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22218) spark shuffle services fails to update secret on application re-attempts

[ https://issues.apache.org/jira/browse/SPARK-22218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22218: Assignee: (was: Apache Spark) > spark shuffle services fails to update secret on

[jira] [Assigned] (SPARK-22218) spark shuffle services fails to update secret on application re-attempts

[ https://issues.apache.org/jira/browse/SPARK-22218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22218: Assignee: Apache Spark > spark shuffle services fails to update secret on application

[jira] [Commented] (SPARK-12394) Support writing out pre-hash-partitioned data and exploit that in join optimizations to avoid shuffle (i.e. bucketing in Hive)

[ https://issues.apache.org/jira/browse/SPARK-12394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195004#comment-16195004 ] Jeetendra Nihalani commented on SPARK-12394: Is this issue resolved? If yes, can someone

[jira] [Assigned] (SPARK-22219) Refector "spark.sql.codegen.comments"

[ https://issues.apache.org/jira/browse/SPARK-22219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22219: Assignee: (was: Apache Spark) > Refector "spark.sql.codegen.comments" >

[jira] [Assigned] (SPARK-22219) Refector "spark.sql.codegen.comments"

[ https://issues.apache.org/jira/browse/SPARK-22219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22219: Assignee: Apache Spark > Refector "spark.sql.codegen.comments" >

[jira] [Commented] (SPARK-22219) Refector "spark.sql.codegen.comments"

[ https://issues.apache.org/jira/browse/SPARK-22219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194966#comment-16194966 ] Apache Spark commented on SPARK-22219: -- User 'kiszk' has created a pull request for this issue:

[jira] [Created] (SPARK-22220) Spark SQL: LATERAL VIEW OUTER null pointer exception with GROUP BY

Dian Fay created SPARK-0: Summary: Spark SQL: LATERAL VIEW OUTER null pointer exception with GROUP BY Key: SPARK-0 URL: https://issues.apache.org/jira/browse/SPARK-0 Project: Spark

[jira] [Created] (SPARK-22219) Refector "spark.sql.codegen.comments"

Kazuaki Ishizaki created SPARK-22219: Summary: Refector "spark.sql.codegen.comments" Key: SPARK-22219 URL: https://issues.apache.org/jira/browse/SPARK-22219 Project: Spark Issue Type:

[jira] [Commented] (SPARK-19984) ERROR codegen.CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java'

[ https://issues.apache.org/jira/browse/SPARK-19984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194937#comment-16194937 ] Kazuaki Ishizaki commented on SPARK-19984: -- [~JohnSteidley] Here is my program that produced the

[jira] [Updated] (SPARK-22218) spark shuffle services fails to update secret on application re-attempts

[ https://issues.apache.org/jira/browse/SPARK-22218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-22218: -- Description: Running on yarn, If you have any application re-attempts using the spark 2.2

[jira] [Updated] (SPARK-22217) ParquetFileFormat to support arbitrary OutputCommitters

[ https://issues.apache.org/jira/browse/SPARK-22217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-22217: --- Priority: Minor (was: Major) > ParquetFileFormat to support arbitrary OutputCommitters >

[jira] [Commented] (SPARK-22218) spark shuffle services fails to update secret on application re-attempts

[ https://issues.apache.org/jira/browse/SPARK-22218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194803#comment-16194803 ] Thomas Graves commented on SPARK-22218: --- Working on a patch for this. > spark shuffle services

[jira] [Created] (SPARK-22218) spark shuffle services fails to update secret on application re-attempts

Thomas Graves created SPARK-22218: - Summary: spark shuffle services fails to update secret on application re-attempts Key: SPARK-22218 URL: https://issues.apache.org/jira/browse/SPARK-22218 Project:

[jira] [Commented] (SPARK-11697) Spark submit can't load user config file using Standalone cluster

[ https://issues.apache.org/jira/browse/SPARK-11697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194800#comment-16194800 ] troopson commented on SPARK-11697: -- I have same problem and take for a while to figure it out. At last I

[jira] [Assigned] (SPARK-22217) ParquetFileFormat to support arbitrary OutputCommitters

[ https://issues.apache.org/jira/browse/SPARK-22217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22217: Assignee: (was: Apache Spark) > ParquetFileFormat to support arbitrary

[jira] [Commented] (SPARK-22217) ParquetFileFormat to support arbitrary OutputCommitters

[ https://issues.apache.org/jira/browse/SPARK-22217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194791#comment-16194791 ] Apache Spark commented on SPARK-22217: -- User 'steveloughran' has created a pull request for this

[jira] [Assigned] (SPARK-22217) ParquetFileFormat to support arbitrary OutputCommitters

[ https://issues.apache.org/jira/browse/SPARK-22217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22217: Assignee: Apache Spark > ParquetFileFormat to support arbitrary OutputCommitters >

[jira] [Created] (SPARK-22217) ParquetFileFormat to support arbitrary OutputCommitters if parquet.enable.summary-metadata is false

Steve Loughran created SPARK-22217: -- Summary: ParquetFileFormat to support arbitrary OutputCommitters if parquet.enable.summary-metadata is false Key: SPARK-22217 URL:

[jira] [Updated] (SPARK-22217) ParquetFileFormat to support arbitrary OutputCommitters

[ https://issues.apache.org/jira/browse/SPARK-22217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-22217: --- Summary: ParquetFileFormat to support arbitrary OutputCommitters (was: ParquetFileFormat to

[jira] [Comment Edited] (SPARK-22216) Improving PySpark/Pandas interoperability

[ https://issues.apache.org/jira/browse/SPARK-22216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194739#comment-16194739 ] Li Jin edited comment on SPARK-22216 at 10/6/17 3:27 PM: - cc [~cloud_fan]

[jira] [Comment Edited] (SPARK-22216) Improving PySpark/Pandas interoperability

[ https://issues.apache.org/jira/browse/SPARK-22216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194739#comment-16194739 ] Li Jin edited comment on SPARK-22216 at 10/6/17 3:27 PM: - cc [~cloud_fan]

[jira] [Commented] (SPARK-22216) Improving PySpark/Pandas interoperability

[ https://issues.apache.org/jira/browse/SPARK-22216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194739#comment-16194739 ] Li Jin commented on SPARK-22216: cc [~cloud_fan] [~rxin] I took the liberty and created this master

[jira] [Updated] (SPARK-22216) Improving PySpark/Pandas interoperability

[ https://issues.apache.org/jira/browse/SPARK-22216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Jin updated SPARK-22216: --- Summary: Improving PySpark/Pandas interoperability (was: Improving PySpark/Pandas interop) > Improving

[jira] [Updated] (SPARK-22216) Improving PySpark/Pandas interop

[ https://issues.apache.org/jira/browse/SPARK-22216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Jin updated SPARK-22216: --- Summary: Improving PySpark/Pandas interop (was: PySpark/Pandas interop umbrella ) > Improving

[jira] [Created] (SPARK-22216) PySpark/Pandas interop umbrella

Li Jin created SPARK-22216: -- Summary: PySpark/Pandas interop umbrella Key: SPARK-22216 URL: https://issues.apache.org/jira/browse/SPARK-22216 Project: Spark Issue Type: Umbrella

[jira] [Assigned] (SPARK-21877) Windows command script can not handle quotes in parameter

[ https://issues.apache.org/jira/browse/SPARK-21877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-21877: Assignee: Xiaokai Zhao > Windows command script can not handle quotes in parameter >

[jira] [Commented] (SPARK-22179) percentile_approx should choose the first element if it already reaches the percentage

[ https://issues.apache.org/jira/browse/SPARK-22179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194680#comment-16194680 ] Zhenhua Wang commented on SPARK-22179: -- [~srowen] Yes, I planned to close this after the new issue

[jira] [Updated] (SPARK-21877) Windows command script can not handle quotes in parameter

[ https://issues.apache.org/jira/browse/SPARK-21877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-21877: - Priority: Minor (was: Major) > Windows command script can not handle quotes in parameter >

[jira] [Resolved] (SPARK-21877) Windows command script can not handle quotes in parameter

[ https://issues.apache.org/jira/browse/SPARK-21877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-21877. -- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19090

[jira] [Assigned] (SPARK-22215) Add a configuration parameter to set max size for generated classes

[ https://issues.apache.org/jira/browse/SPARK-22215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22215: Assignee: (was: Apache Spark) > Add a configuration parameter to set max size for

[jira] [Assigned] (SPARK-22215) Add a configuration parameter to set max size for generated classes

[ https://issues.apache.org/jira/browse/SPARK-22215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22215: Assignee: Apache Spark > Add a configuration parameter to set max size for generated

[jira] [Commented] (SPARK-22215) Add a configuration parameter to set max size for generated classes

[ https://issues.apache.org/jira/browse/SPARK-22215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194623#comment-16194623 ] Apache Spark commented on SPARK-22215: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Created] (SPARK-22215) Add a configuration parameter to set max size for generated classes

Marco Gaido created SPARK-22215: --- Summary: Add a configuration parameter to set max size for generated classes Key: SPARK-22215 URL: https://issues.apache.org/jira/browse/SPARK-22215 Project: Spark

[jira] [Comment Edited] (SPARK-20712) [SPARK 2.1 REGRESSION][SQL] Spark can't read Hive table when column type has length greater than 4000 bytes

[ https://issues.apache.org/jira/browse/SPARK-20712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194542#comment-16194542 ] Gaurav Shah edited comment on SPARK-20712 at 10/6/17 12:38 PM: --- [~maver1ck]

[jira] [Commented] (SPARK-20712) [SPARK 2.1 REGRESSION][SQL] Spark can't read Hive table when column type has length greater than 4000 bytes

[ https://issues.apache.org/jira/browse/SPARK-20712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194542#comment-16194542 ] Gaurav Shah commented on SPARK-20712: - [~maver1ck] I am facing the same issue, I tried executing the

[jira] [Updated] (SPARK-22209) PySpark does not recognize imports from submodules

[ https://issues.apache.org/jira/browse/SPARK-22209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-22209: - Affects Version/s: 2.3.0 > PySpark does not recognize imports from submodules >

[jira] [Commented] (SPARK-22214) Refactor the list hive partitions code

[ https://issues.apache.org/jira/browse/SPARK-22214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194467#comment-16194467 ] Apache Spark commented on SPARK-22214: -- User 'jiangxb1987' has created a pull request for this

[jira] [Assigned] (SPARK-22214) Refactor the list hive partitions code

[ https://issues.apache.org/jira/browse/SPARK-22214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22214: Assignee: (was: Apache Spark) > Refactor the list hive partitions code >

[jira] [Assigned] (SPARK-22214) Refactor the list hive partitions code

[ https://issues.apache.org/jira/browse/SPARK-22214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22214: Assignee: Apache Spark > Refactor the list hive partitions code >

[jira] [Created] (SPARK-22214) Refactor the list hive partitions code

Jiang Xingbo created SPARK-22214: Summary: Refactor the list hive partitions code Key: SPARK-22214 URL: https://issues.apache.org/jira/browse/SPARK-22214 Project: Spark Issue Type:

[jira] [Created] (SPARK-22213) Spark to detect slow executors on nodes with problematic hardware

Oleksandr Konopko created SPARK-22213: - Summary: Spark to detect slow executors on nodes with problematic hardware Key: SPARK-22213 URL: https://issues.apache.org/jira/browse/SPARK-22213 Project:

[jira] [Commented] (SPARK-22212) Some SQL functions in Python fail with string column name

[ https://issues.apache.org/jira/browse/SPARK-22212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194433#comment-16194433 ] Apache Spark commented on SPARK-22212: -- User 'jsnowacki' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22212) Some SQL functions in Python fail with string column name

[ https://issues.apache.org/jira/browse/SPARK-22212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22212: Assignee: Apache Spark > Some SQL functions in Python fail with string column name >

[jira] [Assigned] (SPARK-22212) Some SQL functions in Python fail with string column name

[ https://issues.apache.org/jira/browse/SPARK-22212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22212: Assignee: (was: Apache Spark) > Some SQL functions in Python fail with string column

[jira] [Resolved] (SPARK-21999) ConcurrentModificationException - Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-21999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved SPARK-21999. Resolution: Won't Fix > ConcurrentModificationException - Spark Streaming >

[jira] [Commented] (SPARK-21999) ConcurrentModificationException - Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-21999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194332#comment-16194332 ] Steve Loughran commented on SPARK-21999: Telling a project "their design is wrong" and expecting

[jira] [Created] (SPARK-22212) Some SQL functions in Python fail with string column name

Jakub Nowacki created SPARK-22212: - Summary: Some SQL functions in Python fail with string column name Key: SPARK-22212 URL: https://issues.apache.org/jira/browse/SPARK-22212 Project: Spark

[jira] [Comment Edited] (SPARK-22195) Add cosine similarity to org.apache.spark.ml.linalg.Vectors

[ https://issues.apache.org/jira/browse/SPARK-22195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193844#comment-16193844 ] yuhao yang edited comment on SPARK-22195 at 10/6/17 7:33 AM: - Thanks for the

[jira] [Updated] (SPARK-22187) Update unsaferow format for saved state such that we can set timeouts when state is null

[ https://issues.apache.org/jira/browse/SPARK-22187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-22187: -- Fix Version/s: (was: 3.0.0) 2.3.0 > Update unsaferow format for saved state

[jira] [Commented] (SPARK-22163) Design Issue of Spark Streaming that Causes Random Run-time Exception

[ https://issues.apache.org/jira/browse/SPARK-22163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194228#comment-16194228 ] Sean Owen commented on SPARK-22163: --- This is, at least, a little more specific. The problem is this:

[jira] [Commented] (SPARK-16996) Hive ACID delta files not seen

[ https://issues.apache.org/jira/browse/SPARK-16996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194200#comment-16194200 ] Sindhu Subhas commented on SPARK-16996: --- Hive ACID+Spark is not a supported as of now. Feature is