[jira] [Updated] (SPARK-29932) lint-r should do non-zero exit in case of errors

[ https://issues.apache.org/jira/browse/SPARK-29932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29932: -- Component/s: SparkR > lint-r should do non-zero exit in case of errors >

[jira] [Updated] (SPARK-29932) lint-r should do non-zero exit in case of errors

[ https://issues.apache.org/jira/browse/SPARK-29932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29932: -- Summary: lint-r should do non-zero exit in case of errors (was: lint-r should do non-zero

[jira] [Updated] (SPARK-29932) lint-r should do non-zero exit if there is no R installation

[ https://issues.apache.org/jira/browse/SPARK-29932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29932: -- Summary: lint-r should do non-zero exit if there is no R installation (was: lint-r should do

[jira] [Updated] (SPARK-29932) lint-r should do non-zero exit if there is no R instation

[ https://issues.apache.org/jira/browse/SPARK-29932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29932: -- Summary: lint-r should do non-zero exit if there is no R instation (was: lint-r should do

[jira] [Created] (SPARK-29932) lint-r should do non-zero exit in case of error

Dongjoon Hyun created SPARK-29932: - Summary: lint-r should do non-zero exit in case of error Key: SPARK-29932 URL: https://issues.apache.org/jira/browse/SPARK-29932 Project: Spark Issue

[jira] [Assigned] (SPARK-29858) ALTER DATABASE (SET DBPROPERTIES) should look up catalog like v2 commands

[ https://issues.apache.org/jira/browse/SPARK-29858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29858: - Assignee: Hu Fuwang > ALTER DATABASE (SET DBPROPERTIES) should look up catalog like v2

[jira] [Resolved] (SPARK-29858) ALTER DATABASE (SET DBPROPERTIES) should look up catalog like v2 commands

[ https://issues.apache.org/jira/browse/SPARK-29858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29858. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26551

[jira] [Resolved] (SPARK-29378) Make AppVeyor's SparkR with Arrow tests compatible with Arrow R 0.15

[ https://issues.apache.org/jira/browse/SPARK-29378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29378. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26555

[jira] [Updated] (SPARK-29378) Upgrade SparkR to use Arrow 0.15 API

[ https://issues.apache.org/jira/browse/SPARK-29378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29378: -- Issue Type: Improvement (was: Test) > Upgrade SparkR to use Arrow 0.15 API >

[jira] [Updated] (SPARK-29378) Upgrade SparkR to use Arrow 0.15 API

[ https://issues.apache.org/jira/browse/SPARK-29378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29378: -- Summary: Upgrade SparkR to use Arrow 0.15 API (was: Make AppVeyor's SparkR with Arrow tests

[jira] [Assigned] (SPARK-29378) Make AppVeyor's SparkR with Arrow tests compatible with Arrow R 0.15

[ https://issues.apache.org/jira/browse/SPARK-29378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29378: - Assignee: Dongjoon Hyun > Make AppVeyor's SparkR with Arrow tests compatible with

[jira] [Updated] (SPARK-29924) Document Arrow requirement in JDK9+

[ https://issues.apache.org/jira/browse/SPARK-29924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29924: -- Description: At least, we need to mention `io.netty.tryReflectionSetAccessible=true` is

[jira] [Resolved] (SPARK-29928) Check parsing timestamps up to microsecond precision by JSON/CSV datasource

[ https://issues.apache.org/jira/browse/SPARK-29928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29928. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26558

[jira] [Assigned] (SPARK-29928) Check parsing timestamps up to microsecond precision by JSON/CSV datasource

[ https://issues.apache.org/jira/browse/SPARK-29928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29928: - Assignee: Maxim Gekk > Check parsing timestamps up to microsecond precision by

[jira] [Commented] (SPARK-29890) Unable to fill na with 0 with duplicate columns

[ https://issues.apache.org/jira/browse/SPARK-29890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975853#comment-16975853 ] Terry Kim commented on SPARK-29890: --- {code:java} scala> p1.join(p2, Seq("nums")).printSchema root |--

[jira] [Commented] (SPARK-29931) Declare all SQL legacy configs as will be removed in Spark 4.0

[ https://issues.apache.org/jira/browse/SPARK-29931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975851#comment-16975851 ] Sean R. Owen commented on SPARK-29931: -- I think it's OK to deprecate them if they're legacy. I

[jira] [Updated] (SPARK-29906) Reading of csv file fails with adaptive execution turned on

[ https://issues.apache.org/jira/browse/SPARK-29906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-29906: -- Priority: Minor (was: Major) > Reading of csv file fails with adaptive execution turned on >

[jira] [Commented] (SPARK-29931) Declare all SQL legacy configs as will be removed in Spark 4.0

[ https://issues.apache.org/jira/browse/SPARK-29931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975813#comment-16975813 ] Maxim Gekk commented on SPARK-29931: [~rxin] [~lixiao] [~srowen] [~dongjoon] [~cloud_fan]

[jira] [Created] (SPARK-29931) Declare all SQL legacy configs as will be removed in Spark 4.0

Maxim Gekk created SPARK-29931: -- Summary: Declare all SQL legacy configs as will be removed in Spark 4.0 Key: SPARK-29931 URL: https://issues.apache.org/jira/browse/SPARK-29931 Project: Spark

[jira] [Resolved] (SPARK-29871) Flaky test: ImageFileFormatTest.test_read_images

[ https://issues.apache.org/jira/browse/SPARK-29871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29871. -- Resolution: Invalid You're not really providing any info here. We aren't observing the

[jira] [Commented] (SPARK-29830) PySpark.context.Sparkcontext.binaryfiles improved memory with buffer

[ https://issues.apache.org/jira/browse/SPARK-29830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975809#comment-16975809 ] Sean R. Owen commented on SPARK-29830: -- I don't know how you're going to get a stream from the JVM

[jira] [Commented] (SPARK-29903) Add documentation for recursiveFileLookup

[ https://issues.apache.org/jira/browse/SPARK-29903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975807#comment-16975807 ] Sean R. Owen commented on SPARK-29903: -- Sure, want to open a PR? > Add documentation for

[jira] [Updated] (SPARK-29930) Remove SQL configs declared to be removed in Spark 3.0

[ https://issues.apache.org/jira/browse/SPARK-29930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-29930: - Priority: Minor (was: Major) > Remove SQL configs declared to be removed in Spark 3.0 >

[jira] [Resolved] (SPARK-29878) Improper cache strategies in GraphX

[ https://issues.apache.org/jira/browse/SPARK-29878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29878. -- Resolution: Duplicate > Improper cache strategies in GraphX >

[jira] [Resolved] (SPARK-28781) Unneccesary persist in PeriodicCheckpointer.update()

[ https://issues.apache.org/jira/browse/SPARK-28781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-28781. -- Resolution: Not A Problem I think the point of this class is to manage RDDs that depend on

[jira] [Created] (SPARK-29930) Remove SQL configs declared to be removed in Spark 3.0

Maxim Gekk created SPARK-29930: -- Summary: Remove SQL configs declared to be removed in Spark 3.0 Key: SPARK-29930 URL: https://issues.apache.org/jira/browse/SPARK-29930 Project: Spark Issue

[jira] [Resolved] (SPARK-29827) Wrong persist strategy in mllib.clustering.BisectingKMeans.run

[ https://issues.apache.org/jira/browse/SPARK-29827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29827. -- Resolution: Duplicate Same general answer - it's not clear that persisting is a win here.

[jira] [Resolved] (SPARK-29856) Conditional unnecessary persist on RDDs in ML algorithms

[ https://issues.apache.org/jira/browse/SPARK-29856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29856. -- Resolution: Duplicate > Conditional unnecessary persist on RDDs in ML algorithms >

[jira] [Commented] (SPARK-29810) Missing persist on retaggedInput in RandomForest.run()

[ https://issues.apache.org/jira/browse/SPARK-29810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975800#comment-16975800 ] Sean R. Owen commented on SPARK-29810: -- Generally speaking, it's not necessarily true that you want

[jira] [Commented] (SPARK-29832) Unnecessary persist on instances in ml.regression.IsotonicRegression.fit

[ https://issues.apache.org/jira/browse/SPARK-29832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975799#comment-16975799 ] Sean R. Owen commented on SPARK-29832: -- [~spark_cachecheck] some of these may be valid, but a lot

[jira] [Resolved] (SPARK-29760) Document VALUES statement in SQL Reference.

[ https://issues.apache.org/jira/browse/SPARK-29760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29760. -- Resolution: Won't Fix > Document VALUES statement in SQL Reference. >

[jira] [Resolved] (SPARK-29765) Monitoring UI throws IndexOutOfBoundsException when accessing metrics of attempt in stage

[ https://issues.apache.org/jira/browse/SPARK-29765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29765. -- Resolution: Not A Problem > Monitoring UI throws IndexOutOfBoundsException when accessing

[jira] [Created] (SPARK-29929) Allow V2 Datasources to require a data distribution

Andrew K Long created SPARK-29929: - Summary: Allow V2 Datasources to require a data distribution Key: SPARK-29929 URL: https://issues.apache.org/jira/browse/SPARK-29929 Project: Spark Issue

[jira] [Resolved] (SPARK-29476) Add tooltip information for Thread Dump links and Thread details table columns in Executors Tab

[ https://issues.apache.org/jira/browse/SPARK-29476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29476. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 26386

[jira] [Updated] (SPARK-29476) Add tooltip information for Thread Dump links and Thread details table columns in Executors Tab

[ https://issues.apache.org/jira/browse/SPARK-29476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-29476: - Fix Version/s: (was: 3.1.0) 3.0.0 > Add tooltip information for Thread

[jira] [Assigned] (SPARK-29476) Add tooltip information for Thread Dump links and Thread details table columns in Executors Tab

[ https://issues.apache.org/jira/browse/SPARK-29476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-29476: Assignee: pavithra ramachandran > Add tooltip information for Thread Dump links and

[jira] [Commented] (SPARK-22236) CSV I/O: does not respect RFC 4180

[ https://issues.apache.org/jira/browse/SPARK-22236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975776#comment-16975776 ] Santhosh commented on SPARK-22236: -- The code mentioned above  spark.read.option('escape',

[jira] [Created] (SPARK-29928) Check parsing timestamps up to microsecond precision by JSON/CSV datasource

Maxim Gekk created SPARK-29928: -- Summary: Check parsing timestamps up to microsecond precision by JSON/CSV datasource Key: SPARK-29928 URL: https://issues.apache.org/jira/browse/SPARK-29928 Project:

[jira] [Resolved] (SPARK-29818) Missing persist on RDD

[ https://issues.apache.org/jira/browse/SPARK-29818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29818. -- Fix Version/s: (was: 3.0.0) Resolution: Not A Problem > Missing persist on RDD >

[jira] [Commented] (SPARK-29890) Unable to fill na with 0 with duplicate columns

[ https://issues.apache.org/jira/browse/SPARK-29890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975733#comment-16975733 ] sandeshyapuram commented on SPARK-29890: [~imback82] This happens even for a normal join:

[jira] [Commented] (SPARK-29606) Improve EliminateOuterJoin performance

[ https://issues.apache.org/jira/browse/SPARK-29606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975732#comment-16975732 ] Yuming Wang commented on SPARK-29606: - Our production(Spark 2.3): {noformat} === Metrics of

[jira] [Updated] (SPARK-29904) Parse timestamps in microsecond precision by JSON/CSV datasources

[ https://issues.apache.org/jira/browse/SPARK-29904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-29904: --- Affects Version/s: 2.4.0 2.4.1 2.4.2

[jira] [Commented] (SPARK-29927) Parse timestamps in microsecond precision by `to_timestamp`, `to_unix_timestamp`, `unix_timestamp`

[ https://issues.apache.org/jira/browse/SPARK-29927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975697#comment-16975697 ] Maxim Gekk commented on SPARK-29927: [~cloud_fan] WDYT, does it make sense to change the functions

[jira] [Created] (SPARK-29927) Parse timestamps in microsecond precision by `to_timestamp`, `to_unix_timestamp`, `unix_timestamp`

Maxim Gekk created SPARK-29927: -- Summary: Parse timestamps in microsecond precision by `to_timestamp`, `to_unix_timestamp`, `unix_timestamp` Key: SPARK-29927 URL: https://issues.apache.org/jira/browse/SPARK-29927

[jira] [Updated] (SPARK-29923) Set `io.netty.tryReflectionSetAccessible` for Arrow on JDK9+

[ https://issues.apache.org/jira/browse/SPARK-29923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-29923: - Docs Text: Spark applications running on JDK 9 or later must set the system property

[jira] [Comment Edited] (SPARK-29925) Maven Build fails with Hadoop Version 3.2.0

[ https://issues.apache.org/jira/browse/SPARK-29925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975681#comment-16975681 ] Yuming Wang edited comment on SPARK-29925 at 11/16/19 11:50 AM: You

[jira] [Resolved] (SPARK-29925) Maven Build fails with Hadoop Version 3.2.0

[ https://issues.apache.org/jira/browse/SPARK-29925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-29925. - Resolution: Invalid > Maven Build fails with Hadoop Version 3.2.0 >

[jira] [Commented] (SPARK-29925) Maven Build fails with Hadoop Version 3.2.0

[ https://issues.apache.org/jira/browse/SPARK-29925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975681#comment-16975681 ] Yuming Wang commented on SPARK-29925: - You should build with {{hadoop-3.2}} profile.

[jira] [Created] (SPARK-29926) interval `1. second` should be invalid as PostgreSQL

Kent Yao created SPARK-29926: Summary: interval `1. second` should be invalid as PostgreSQL Key: SPARK-29926 URL: https://issues.apache.org/jira/browse/SPARK-29926 Project: Spark Issue Type:

[jira] [Commented] (SPARK-29926) interval `1. second` should be invalid as PostgreSQL

[ https://issues.apache.org/jira/browse/SPARK-29926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975657#comment-16975657 ] Kent Yao commented on SPARK-29926: -- working on this > interval `1. second` should be invalid as

[jira] [Assigned] (SPARK-29807) Rename "spark.sql.ansi.enabled" to "spark.sql.dialect.spark.ansi.enabled"

[ https://issues.apache.org/jira/browse/SPARK-29807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-29807: --- Assignee: Yuanjian Li > Rename "spark.sql.ansi.enabled" to

[jira] [Updated] (SPARK-29925) Maven Build fails with Hadoop Version 3.2.0

[ https://issues.apache.org/jira/browse/SPARK-29925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Douglas Colkitt updated SPARK-29925: Description: Build fails at Spark Core stage when using Maven with specified Hadoop

[jira] [Updated] (SPARK-29925) Maven Build fails with Hadoop Version 3.2.0

[ https://issues.apache.org/jira/browse/SPARK-29925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Douglas Colkitt updated SPARK-29925: Description: Build fails at Spark Core stage when using Maven with specified Hadoop Cloud

[jira] [Created] (SPARK-29925) Maven Build fails with flag: -Phadoop-cloud

Douglas Colkitt created SPARK-29925: --- Summary: Maven Build fails with flag: -Phadoop-cloud Key: SPARK-29925 URL: https://issues.apache.org/jira/browse/SPARK-29925 Project: Spark Issue