[jira] [Updated] (SPARK-27894) PySpark streaming transform RDD join not works when checkpoint enabled

2019-05-30 Thread Jeffrey(Xilang) Yan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey(Xilang) Yan updated SPARK-27894: Description: In PySpark Steaming, if checkpoint enabled and there is a transform-j

[jira] [Updated] (SPARK-27894) PySpark streaming transform RDD join not works when checkpoint enabled

2019-05-30 Thread Jeffrey(Xilang) Yan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey(Xilang) Yan updated SPARK-27894: Description: In PySpark Steaming, if checkpoint enabled and there is a transform-j

[jira] [Created] (SPARK-27894) PySpark streaming transform RDD join not works when checkpoint enabled

2019-05-30 Thread Jeffrey(Xilang) Yan (JIRA)
Jeffrey(Xilang) Yan created SPARK-27894: --- Summary: PySpark streaming transform RDD join not works when checkpoint enabled Key: SPARK-27894 URL: https://issues.apache.org/jira/browse/SPARK-27894

[jira] [Assigned] (SPARK-27893) Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files

2019-05-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27893: Assignee: Apache Spark > Create an integrated test base for Python, Scalar Pandas, Scala

[jira] [Assigned] (SPARK-27893) Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files

2019-05-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27893: Assignee: (was: Apache Spark) > Create an integrated test base for Python, Scalar Pan

[jira] [Created] (SPARK-27893) Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files

2019-05-30 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-27893: Summary: Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files Key: SPARK-27893 URL: https://issues.apache.org/jira/browse/SPARK-27893 Proj

[jira] [Comment Edited] (SPARK-24815) Structured Streaming should support dynamic allocation

2019-05-30 Thread Karthik Palaniappan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852541#comment-16852541 ] Karthik Palaniappan edited comment on SPARK-24815 at 5/31/19 2:26 AM:

[jira] [Commented] (SPARK-27891) Long running spark jobs fail because of HDFS delegation token expires

2019-05-30 Thread hemshankar sahu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852573#comment-16852573 ] hemshankar sahu commented on SPARK-27891: - Sure I'll provide for 2.3.1 in some t

[jira] [Updated] (SPARK-27876) Split large shuffle partition to multi-segments to enable transfer oversize shuffle partition block.

2019-05-30 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27876: Affects Version/s: (was: 3.1.0) (was: 2.4.3) 2.3.2 >

[jira] [Commented] (SPARK-24815) Structured Streaming should support dynamic allocation

2019-05-30 Thread Karthik Palaniappan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852541#comment-16852541 ] Karthik Palaniappan commented on SPARK-24815: - I was starting to update the

[jira] [Updated] (SPARK-24815) Structured Streaming should support dynamic allocation

2019-05-30 Thread Karthik Palaniappan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Palaniappan updated SPARK-24815: Description: For batch jobs, dynamic allocation is very useful for adding and remo

[jira] [Commented] (SPARK-27884) Deprecate Python 2 support in Spark 3.0

2019-05-30 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852508#comment-16852508 ] Hyukjin Kwon commented on SPARK-27884: -- +1 > Deprecate Python 2 support in Spark 3

[jira] [Assigned] (SPARK-27862) Upgrade json4s-jackson to 3.6.6

2019-05-30 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-27862: - Assignee: Izek Greenfield > Upgrade json4s-jackson to 3.6.6 > --- >

[jira] [Resolved] (SPARK-27862) Upgrade json4s-jackson to 3.6.6

2019-05-30 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27862. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24736 [https://github.c

[jira] [Updated] (SPARK-27892) Saving/loading stages in PipelineModel should be parallel

2019-05-30 Thread Jason Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Wang updated SPARK-27892: --- Description: When a PipelineModel is saved/loaded, all the stages are saved/loaded sequentially. Wh

[jira] [Created] (SPARK-27892) Saving/loading stages in PipelineModel should be parallel

2019-05-30 Thread Jason Wang (JIRA)
Jason Wang created SPARK-27892: -- Summary: Saving/loading stages in PipelineModel should be parallel Key: SPARK-27892 URL: https://issues.apache.org/jira/browse/SPARK-27892 Project: Spark Issue T

[jira] [Assigned] (SPARK-27684) Reduce ScalaUDF conversion overheads for primitives

2019-05-30 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-27684: -- Assignee: Marco Gaido > Reduce ScalaUDF conversion overheads for primitives > ---

[jira] [Resolved] (SPARK-27684) Reduce ScalaUDF conversion overheads for primitives

2019-05-30 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-27684. Resolution: Fixed Fix Version/s: 3.0.0 Fixed for 3.0 in https://github.com/apache/spark/pul

[jira] [Assigned] (SPARK-27890) Improve SQL parser error message when missing backquotes for identifiers with hyphens

2019-05-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27890: Assignee: Apache Spark > Improve SQL parser error message when missing backquotes for ide

[jira] [Assigned] (SPARK-27890) Improve SQL parser error message when missing backquotes for identifiers with hyphens

2019-05-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27890: Assignee: (was: Apache Spark) > Improve SQL parser error message when missing backquo

[jira] [Updated] (SPARK-27772) SQLTestUtils Refactoring

2019-05-30 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27772: -- Component/s: Tests > SQLTestUtils Refactoring > > > K

[jira] [Commented] (SPARK-27891) Long running spark jobs fail because of HDFS delegation token expires

2019-05-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852420#comment-16852420 ] Marcelo Vanzin commented on SPARK-27891: Ok, the updated logs show the issue. Bu

[jira] [Updated] (SPARK-27891) Long running spark jobs fail because of HDFS delegation token expires

2019-05-30 Thread hemshankar sahu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemshankar sahu updated SPARK-27891: Description: When the spark job runs on a secured cluster for longer then time that is me

[jira] [Updated] (SPARK-27891) Long running spark jobs fail because of HDFS delegation token expires

2019-05-30 Thread hemshankar sahu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemshankar sahu updated SPARK-27891: Attachment: application_1559242207407_0001.log > Long running spark jobs fail because of H

[jira] [Commented] (SPARK-27891) Long running spark jobs fail because of HDFS delegation token expires

2019-05-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852408#comment-16852408 ] Marcelo Vanzin commented on SPARK-27891: {{container_e48_1559242207407_0001_02_0

[jira] [Updated] (SPARK-27891) Long running spark jobs fail because of HDFS delegation token expires

2019-05-30 Thread hemshankar sahu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemshankar sahu updated SPARK-27891: Description: When the spark job runs on a secured cluster for longer then time that is me

[jira] [Reopened] (SPARK-27891) Long running spark jobs fail because of HDFS delegation token expires

2019-05-30 Thread hemshankar sahu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemshankar sahu reopened SPARK-27891: - We used following command to submit the job bin/spark-submit --principal acekrbuser --keyta

[jira] [Comment Edited] (SPARK-27812) kubernetes client import non-daemon thread which block jvm exit.

2019-05-30 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852402#comment-16852402 ] Dongjoon Hyun edited comment on SPARK-27812 at 5/30/19 10:06 PM: -

[jira] [Commented] (SPARK-27812) kubernetes client import non-daemon thread which block jvm exit.

2019-05-30 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852402#comment-16852402 ] Dongjoon Hyun commented on SPARK-27812: --- Thank you for reporting, [~Andrew HUALI]

[jira] [Issue Comment Deleted] (SPARK-22151) PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly

2019-05-30 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruslan Dautkhanov updated SPARK-22151: -- Comment: was deleted (was: Is there is a workaround for this in Apache Livy? We're st

[jira] [Resolved] (SPARK-27891) Long running spark jobs fail because of HDFS delegation token expires

2019-05-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27891. Resolution: Not A Problem You have to provide a keytab for Spark for this to work. That's

[jira] [Created] (SPARK-27891) Long running spark jobs fail because of HDFS delegation token expires

2019-05-30 Thread hemshankar sahu (JIRA)
hemshankar sahu created SPARK-27891: --- Summary: Long running spark jobs fail because of HDFS delegation token expires Key: SPARK-27891 URL: https://issues.apache.org/jira/browse/SPARK-27891 Project:

[jira] [Resolved] (SPARK-27361) YARN support for GPU-aware scheduling

2019-05-30 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27361. --- Resolution: Fixed Fix Version/s: 3.0.0 all the subtasks are finished and the parts of

[jira] [Commented] (SPARK-22151) PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly

2019-05-30 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852381#comment-16852381 ] Ruslan Dautkhanov commented on SPARK-22151: --- Is there is a workaround for this

[jira] [Created] (SPARK-27890) Improve SQL parser error message when missing backquotes for identifiers with hyphens

2019-05-30 Thread Yesheng Ma (JIRA)
Yesheng Ma created SPARK-27890: -- Summary: Improve SQL parser error message when missing backquotes for identifiers with hyphens Key: SPARK-27890 URL: https://issues.apache.org/jira/browse/SPARK-27890 Pro

[jira] [Assigned] (SPARK-27872) Driver and executors use a different service account breaking pull secrets

2019-05-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27872: Assignee: Apache Spark > Driver and executors use a different service account breaking pu

[jira] [Assigned] (SPARK-27872) Driver and executors use a different service account breaking pull secrets

2019-05-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27872: Assignee: (was: Apache Spark) > Driver and executors use a different service account

[jira] [Commented] (SPARK-24149) Automatic namespaces discovery in HDFS federation

2019-05-30 Thread Dhruve Ashar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852365#comment-16852365 ] Dhruve Ashar commented on SPARK-24149: -- IMHO having a consistent way to reason abou

[jira] [Resolved] (SPARK-27773) Add shuffle service metric for number of exceptions caught in ExternalShuffleBlockHandler

2019-05-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27773. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24645 [https:

[jira] [Assigned] (SPARK-27773) Add shuffle service metric for number of exceptions caught in ExternalShuffleBlockHandler

2019-05-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27773: -- Assignee: Steven Rand > Add shuffle service metric for number of exceptions caught in

[jira] [Commented] (SPARK-27812) kubernetes client import non-daemon thread which block jvm exit.

2019-05-30 Thread Igor Calabria (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852342#comment-16852342 ] Igor Calabria commented on SPARK-27812: --- I believe this was introduced when `kuber

[jira] [Resolved] (SPARK-27378) spark-submit requests GPUs in YARN mode

2019-05-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27378. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24634 [https:

[jira] [Assigned] (SPARK-27378) spark-submit requests GPUs in YARN mode

2019-05-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27378: -- Assignee: Thomas Graves > spark-submit requests GPUs in YARN mode > -

[jira] [Commented] (SPARK-27886) Add Apache Spark project to https://python3statement.org/

2019-05-30 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852276#comment-16852276 ] Xiangrui Meng commented on SPARK-27886: --- By "at the end of year", you mean year 20

[jira] [Commented] (SPARK-27798) from_avro can modify variables in other rows in local mode

2019-05-30 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852268#comment-16852268 ] Dongjoon Hyun commented on SPARK-27798: --- Thank you so much for the investigation a

[jira] [Commented] (SPARK-27884) Deprecate Python 2 support in Spark 3.0

2019-05-30 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852238#comment-16852238 ] Dongjoon Hyun commented on SPARK-27884: --- Thanks! +1 for this efforts > Deprecate

[jira] [Commented] (SPARK-27886) Add Apache Spark project to https://python3statement.org/

2019-05-30 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852166#comment-16852166 ] Sean Owen commented on SPARK-27886: --- Maybe I misunderstand, but if it's likely Python

[jira] [Commented] (SPARK-27884) Deprecate Python 2 support in Spark 3.0

2019-05-30 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852149#comment-16852149 ] Sean Owen commented on SPARK-27884: --- Yes that looks fine to me. > Deprecate Python 2

[jira] [Commented] (SPARK-27772) SQLTestUtils Refactoring

2019-05-30 Thread William Wong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852148#comment-16852148 ] William Wong commented on SPARK-27772: -- Hi [~hyukjin.kwon], I submitted the PR and

[jira] [Created] (SPARK-27889) Make development scripts under dev/ support Python 3

2019-05-30 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27889: - Summary: Make development scripts under dev/ support Python 3 Key: SPARK-27889 URL: https://issues.apache.org/jira/browse/SPARK-27889 Project: Spark Issue

[jira] [Commented] (SPARK-27648) In Spark2.4 Structured Streaming:The executor storage memory increasing over time

2019-05-30 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852136#comment-16852136 ] Gabor Somogyi commented on SPARK-27648: --- I've made my graphs based on the files yo

[jira] [Updated] (SPARK-27772) SQLTestUtils Refactoring

2019-05-30 Thread William Wong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Wong updated SPARK-27772: - Description: The current `SQLTestUtils` created many `withXXX` utility functions to clean up ta

[jira] [Commented] (SPARK-27772) SQLTestUtils Refactoring

2019-05-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852125#comment-16852125 ] Apache Spark commented on SPARK-27772: -- User 'William1104' has created a pull reque

[jira] [Assigned] (SPARK-27772) SQLTestUtils Refactoring

2019-05-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27772: Assignee: Apache Spark > SQLTestUtils Refactoring > > >

[jira] [Assigned] (SPARK-27772) SQLTestUtils Refactoring

2019-05-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27772: Assignee: (was: Apache Spark) > SQLTestUtils Refactoring > >

[jira] [Commented] (SPARK-27772) SQLTestUtils Refactoring

2019-05-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852124#comment-16852124 ] Apache Spark commented on SPARK-27772: -- User 'William1104' has created a pull reque

[jira] [Updated] (SPARK-27772) SQLTestUtils Refactoring

2019-05-30 Thread William Wong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Wong updated SPARK-27772: - Description: The current `SQLTestUtils` created many `withXXX` utility functions to clean up ta

[jira] [Commented] (SPARK-27785) Introduce .joinWith() overloads for typed inner joins of 3 or more tables

2019-05-30 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852089#comment-16852089 ] Josh Rosen commented on SPARK-27785: I think this might require a little bit more de

[jira] [Updated] (SPARK-27885) Announce deprecation of Python 2 support

2019-05-30 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27885: -- Description: * Draft the message. * Update Spark website and announce deprecation of Python 2

[jira] [Commented] (SPARK-27886) Add Apache Spark project to https://python3statement.org/

2019-05-30 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852088#comment-16852088 ] Xiangrui Meng commented on SPARK-27886: --- cc: [~srowen] [~smilegator] > Add Apache

[jira] [Assigned] (SPARK-27831) Move Hive test jars to maven dependency

2019-05-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27831: Assignee: (was: Apache Spark) > Move Hive test jars to maven dependency > ---

[jira] [Updated] (SPARK-27886) Add Apache Spark project to https://python3statement.org/

2019-05-30 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27886: -- Description: Add Spark to https://python3statement.org/ and indicate our timeline. I reviewed

[jira] [Assigned] (SPARK-27831) Move Hive test jars to maven dependency

2019-05-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27831: Assignee: (was: Apache Spark) > Move Hive test jars to maven dependency > ---

[jira] [Assigned] (SPARK-27831) Move Hive test jars to maven dependency

2019-05-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27831: Assignee: Apache Spark > Move Hive test jars to maven dependency > --

[jira] [Updated] (SPARK-27886) Add Apache Spark project to https://python3statement.org/

2019-05-30 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27886: -- Description: Add Spark to https://python3statement.org/ and indicate our timeline. I reviewed

[jira] [Reopened] (SPARK-27831) Move Hive test jars to maven dependency

2019-05-30 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-27831: --- Assignee: (was: Yuming Wang) > Move Hive test jars to maven dependency > -

[jira] [Commented] (SPARK-27884) Deprecate Python 2 support in Spark 3.0

2019-05-30 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852077#comment-16852077 ] Xiangrui Meng commented on SPARK-27884: --- [~srowen] Could you help review the draft

[jira] [Resolved] (SPARK-27813) DataSourceV2: Add DropTable logical operation

2019-05-30 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27813. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24686 [https://gith

[jira] [Commented] (SPARK-27884) Deprecate Python 2 support in Spark 3.0

2019-05-30 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852071#comment-16852071 ] Xiangrui Meng commented on SPARK-27884: --- Draft message: *Apache Spark's plan for

[jira] [Assigned] (SPARK-27813) DataSourceV2: Add DropTable logical operation

2019-05-30 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-27813: --- Assignee: John Zhuge > DataSourceV2: Add DropTable logical operation >

[jira] [Commented] (SPARK-27798) from_avro can modify variables in other rows in local mode

2019-05-30 Thread Gengliang Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852055#comment-16852055 ] Gengliang Wang commented on SPARK-27798: Also, the issue can be reproduced on la

[jira] [Commented] (SPARK-25557) ORC predicate pushdown for nested fields

2019-05-30 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852053#comment-16852053 ] Dongjoon Hyun commented on SPARK-25557: --- Nope. The nested scheme pruning is indepe

[jira] [Updated] (SPARK-27862) Upgrade json4s-jackson to 3.6.6

2019-05-30 Thread Izek Greenfield (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Izek Greenfield updated SPARK-27862: Summary: Upgrade json4s-jackson to 3.6.6 (was: Upgrade json4s-jackson to 3.6.5) > Upgrad

[jira] [Updated] (SPARK-27884) Deprecate Python 2 support in Spark 3.0

2019-05-30 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27884: -- Description: Officially deprecate Python 2 support in Spark 3.0. (was: Officially deprecate P

[jira] [Updated] (SPARK-27885) Announce deprecation of Python 2 support

2019-05-30 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27885: -- Description: * Draft the message. * Update Spark website and announce deprecation of Python 2

[jira] [Created] (SPARK-27888) Python 2->3 migration guide for PySpark users

2019-05-30 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27888: - Summary: Python 2->3 migration guide for PySpark users Key: SPARK-27888 URL: https://issues.apache.org/jira/browse/SPARK-27888 Project: Spark Issue Type: S

[jira] [Updated] (SPARK-27885) Announce deprecation of Python 2 support

2019-05-30 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27885: -- Summary: Announce deprecation of Python 2 support (was: Update Spark website and put deprecat

[jira] [Updated] (SPARK-27887) Check python version and print deprecation warning if version < 3

2019-05-30 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27887: -- Description: In Spark 3.0, users should see a deprecation warning if they use PySpark with Pyt

[jira] [Created] (SPARK-27887) Check python version and print deprecation warning if version < 3

2019-05-30 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27887: - Summary: Check python version and print deprecation warning if version < 3 Key: SPARK-27887 URL: https://issues.apache.org/jira/browse/SPARK-27887 Project: Spark

[jira] [Updated] (SPARK-27885) Update Spark website and put deprecation warning

2019-05-30 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27885: -- Description: Update Spark website and announce deprecation of Python 2 support in the next maj

[jira] [Created] (SPARK-27886) Add Apache Spark project to https://python3statement.org/

2019-05-30 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27886: - Summary: Add Apache Spark project to https://python3statement.org/ Key: SPARK-27886 URL: https://issues.apache.org/jira/browse/SPARK-27886 Project: Spark I

[jira] [Created] (SPARK-27885) Update Spark website and put deprecation warning

2019-05-30 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27885: - Summary: Update Spark website and put deprecation warning Key: SPARK-27885 URL: https://issues.apache.org/jira/browse/SPARK-27885 Project: Spark Issue Type

[jira] [Created] (SPARK-27884) Deprecate Python 2 support in Spark 3.0

2019-05-30 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27884: - Summary: Deprecate Python 2 support in Spark 3.0 Key: SPARK-27884 URL: https://issues.apache.org/jira/browse/SPARK-27884 Project: Spark Issue Type: Story

[jira] [Commented] (SPARK-27742) Security Support in Sources and Sinks for SS and Batch

2019-05-30 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851979#comment-16851979 ] Gabor Somogyi commented on SPARK-27742: --- {quote}For the user this means better fle

[jira] [Comment Edited] (SPARK-27798) from_avro can modify variables in other rows in local mode

2019-05-30 Thread Gengliang Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851966#comment-16851966 ] Gengliang Wang edited comment on SPARK-27798 at 5/30/19 3:07 PM: -

[jira] [Commented] (SPARK-27798) from_avro can modify variables in other rows in local mode

2019-05-30 Thread Gengliang Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851966#comment-16851966 ] Gengliang Wang commented on SPARK-27798: Turning off the rule "ConvertToLocalRel

[jira] [Closed] (SPARK-27706) Add SQL metrics of numOutputRows for BroadcastExchangeExec

2019-05-30 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-27706. - > Add SQL metrics of numOutputRows for BroadcastExchangeExec > -

[jira] [Comment Edited] (SPARK-27742) Security Support in Sources and Sinks for SS and Batch

2019-05-30 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851924#comment-16851924 ] Stavros Kontopoulos edited comment on SPARK-27742 at 5/30/19 2:58 PM:

[jira] [Comment Edited] (SPARK-27742) Security Support in Sources and Sinks for SS and Batch

2019-05-30 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851924#comment-16851924 ] Stavros Kontopoulos edited comment on SPARK-27742 at 5/30/19 2:57 PM:

[jira] [Comment Edited] (SPARK-27742) Security Support in Sources and Sinks for SS and Batch

2019-05-30 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851924#comment-16851924 ] Stavros Kontopoulos edited comment on SPARK-27742 at 5/30/19 2:55 PM:

[jira] [Comment Edited] (SPARK-27742) Security Support in Sources and Sinks for SS and Batch

2019-05-30 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851924#comment-16851924 ] Stavros Kontopoulos edited comment on SPARK-27742 at 5/30/19 2:55 PM:

[jira] [Comment Edited] (SPARK-27742) Security Support in Sources and Sinks for SS and Batch

2019-05-30 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851924#comment-16851924 ] Stavros Kontopoulos edited comment on SPARK-27742 at 5/30/19 2:53 PM:

[jira] [Comment Edited] (SPARK-27742) Security Support in Sources and Sinks for SS and Batch

2019-05-30 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851924#comment-16851924 ] Stavros Kontopoulos edited comment on SPARK-27742 at 5/30/19 2:53 PM:

[jira] [Commented] (SPARK-27873) Csv reader, adding a corrupt record column causes error if enforceSchema=false

2019-05-30 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851952#comment-16851952 ] Liang-Chi Hsieh commented on SPARK-27873: - I can prepare a PR if Marcin or Hyukj

[jira] [Commented] (SPARK-27873) Csv reader, adding a corrupt record column causes error if enforceSchema=false

2019-05-30 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851948#comment-16851948 ] Liang-Chi Hsieh commented on SPARK-27873: - I guess what Marcin meant is: {code}

[jira] [Updated] (SPARK-27757) Bump Jackson to 2.9.9

2019-05-30 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-27757: -- Priority: Minor (was: Major) > Bump Jackson to 2.9.9 > - > > Key:

[jira] [Assigned] (SPARK-27757) Bump Jackson to 2.9.9

2019-05-30 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-27757: - Assignee: Fokko Driesprong > Bump Jackson to 2.9.9 > - > >

[jira] [Resolved] (SPARK-27757) Bump Jackson to 2.9.9

2019-05-30 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27757. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24646 [https://github.c

[jira] [Assigned] (SPARK-27883) Add aggregates.sql - Part2

2019-05-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27883: Assignee: (was: Apache Spark) > Add aggregates.sql - Part2 >

[jira] [Assigned] (SPARK-27883) Add aggregates.sql - Part2

2019-05-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27883: Assignee: Apache Spark > Add aggregates.sql - Part2 > -- > >

  1   2   >