[jira] [Commented] (SPARK-32702) Update MiMa plugin

2020-08-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184931#comment-17184931 ] Hyukjin Kwon commented on SPARK-32702: -- [~gemelen], you can take a quickest path for you to upgrade

[jira] [Commented] (SPARK-32636) AsyncEventQueue: Exception scala.Some cannot be cast to java.lang.String

2020-08-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184929#comment-17184929 ] Hyukjin Kwon commented on SPARK-32636: -- I think this is likely from the mismatched Jackson version.

[jira] [Commented] (SPARK-32673) Pyspark/cloudpickle.py - no module named 'wfdb'

2020-08-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184928#comment-17184928 ] Hyukjin Kwon commented on SPARK-32673: -- Yeah, it does look specific to Databricks'. It should be

[jira] [Assigned] (SPARK-31936) Implement ScriptTransform in sql/core

2020-08-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-31936: Assignee: angerszhu > Implement ScriptTransform in sql/core >

[jira] [Commented] (SPARK-32692) Support INSERT OVERWRITE DIR cross cluster

2020-08-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184927#comment-17184927 ] Hyukjin Kwon commented on SPARK-32692: -- ping [~angerszhuuu] > Support INSERT OVERWRITE DIR cross

[jira] [Resolved] (SPARK-32694) Pushdown cast to data sources

2020-08-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32694. -- Resolution: Duplicate > Pushdown cast to data sources > - > >

[jira] [Resolved] (SPARK-32697) Direct Date and timestamp format data insertion fails

2020-08-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32697. -- Resolution: Not A Problem > Direct Date and timestamp format data insertion fails >

[jira] [Commented] (SPARK-32697) Direct Date and timestamp format data insertion fails

2020-08-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184925#comment-17184925 ] Hyukjin Kwon commented on SPARK-32697: -- you can explicitly cast. {code} create table test(no

[jira] [Commented] (SPARK-32699) Add percentage of missingness to df.summary()

2020-08-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184924#comment-17184924 ] Hyukjin Kwon commented on SPARK-32699: -- +1 for Sean's. Let's don't add every details into it. It's

[jira] [Resolved] (SPARK-32699) Add percentage of missingness to df.summary()

2020-08-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32699. -- Resolution: Won't Fix > Add percentage of missingness to df.summary() >

[jira] [Resolved] (SPARK-32695) Add 'build' and 'project/build.properties' into cache key of SBT and Zinc

2020-08-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32695. -- Fix Version/s: 3.1.0 2.4.7 3.0.1 Resolution:

[jira] [Assigned] (SPARK-32695) Add 'build' and 'project/build.properties' into cache key of SBT and Zinc

2020-08-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32695: Assignee: Hyukjin Kwon > Add 'build' and 'project/build.properties' into cache key of

[jira] [Resolved] (SPARK-32182) Getting Started - Quickstart

2020-08-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32182. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29491

[jira] [Assigned] (SPARK-32182) Getting Started - Quickstart

2020-08-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32182: Assignee: Hyukjin Kwon > Getting Started - Quickstart > > >

[jira] [Assigned] (SPARK-32204) Binder Integration

2020-08-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32204: Assignee: Hyukjin Kwon > Binder Integration > -- > >

[jira] [Resolved] (SPARK-32204) Binder Integration

2020-08-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32204. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29491

[jira] [Resolved] (SPARK-32700) select from table TABLESAMPLE gives wrong resultset.

2020-08-25 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-32700. -- Resolution: Invalid > select from table TABLESAMPLE gives wrong resultset. >

[jira] [Commented] (SPARK-32700) select from table TABLESAMPLE gives wrong resultset.

2020-08-25 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184883#comment-17184883 ] Takeshi Yamamuro commented on SPARK-32700: -- +1 on the Sean comment. I'll close this. > select

[jira] [Commented] (SPARK-32466) Add support to catch SparkPlan regression base on TPC-DS queries

2020-08-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184856#comment-17184856 ] Apache Spark commented on SPARK-32466: -- User 'Ngone51' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32620) Reset the numPartitions metric when DPP is enabled

2020-08-25 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-32620: --- Assignee: Yuming Wang > Reset the numPartitions metric when DPP is enabled >

[jira] [Resolved] (SPARK-32620) Reset the numPartitions metric when DPP is enabled

2020-08-25 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-32620. - Fix Version/s: 3.1.0 3.0.1 Resolution: Fixed Issue resolved by pull

[jira] [Updated] (SPARK-32659) Fix the data issue of inserted DPP on non-atomic type

2020-08-25 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-32659: Description: DPP has data issue when pruning on non-atomic type. for example: {noformat}

[jira] [Updated] (SPARK-32659) Fix the data issue of inserted DPP on non-atomic type

2020-08-25 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-32659: Labels: correctness (was: ) > Fix the data issue of inserted DPP on non-atomic type >

[jira] [Updated] (SPARK-32659) Fix the data issue of inserted DPP on non-atomic type

2020-08-25 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-32659: Description: DPP has data issue when pruning on non-atomic type. for example: {noformat}

[jira] [Updated] (SPARK-32659) Fix the data issue of inserted DPP on non-atomic type

2020-08-25 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-32659: Description: DPP has data issue when pruning on non-atomic type. for example: {noformat}

[jira] [Updated] (SPARK-32659) Fix the data issue of inserted DPP on non-atomic type

2020-08-25 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-32659: Description: Fix data issue when adding DPP on non-atomic type. for example: ```scala

[jira] [Assigned] (SPARK-32704) Logging plan changes for execution

2020-08-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32704: Assignee: (was: Apache Spark) > Logging plan changes for execution >

[jira] [Commented] (SPARK-32704) Logging plan changes for execution

2020-08-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184824#comment-17184824 ] Apache Spark commented on SPARK-32704: -- User 'maropu' has created a pull request for this issue:

[jira] [Commented] (SPARK-32704) Logging plan changes for execution

2020-08-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184823#comment-17184823 ] Apache Spark commented on SPARK-32704: -- User 'maropu' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32704) Logging plan changes for execution

2020-08-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32704: Assignee: Apache Spark > Logging plan changes for execution >

[jira] [Created] (SPARK-32704) Logging plan changes for execution

2020-08-25 Thread Takeshi Yamamuro (Jira)
Takeshi Yamamuro created SPARK-32704: Summary: Logging plan changes for execution Key: SPARK-32704 URL: https://issues.apache.org/jira/browse/SPARK-32704 Project: Spark Issue Type:

[jira] [Commented] (SPARK-32516) path option is treated differently for 'format("parquet").load(path)' vs. 'parquet(path)'

2020-08-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184772#comment-17184772 ] Apache Spark commented on SPARK-32516: -- User 'imback82' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32703) Enable dictionary filtering for Parquet vectorized reader

2020-08-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32703: Assignee: (was: Apache Spark) > Enable dictionary filtering for Parquet vectorized

[jira] [Assigned] (SPARK-32703) Enable dictionary filtering for Parquet vectorized reader

2020-08-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32703: Assignee: Apache Spark > Enable dictionary filtering for Parquet vectorized reader >

[jira] [Commented] (SPARK-32703) Enable dictionary filtering for Parquet vectorized reader

2020-08-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184755#comment-17184755 ] Apache Spark commented on SPARK-32703: -- User 'sunchao' has created a pull request for this issue:

[jira] [Commented] (SPARK-32701) mapreduce.fileoutputcommitter.algorithm.version default depends on runtime environment

2020-08-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184730#comment-17184730 ] Apache Spark commented on SPARK-32701: -- User 'waleedfateem' has created a pull request for this

[jira] [Assigned] (SPARK-32701) mapreduce.fileoutputcommitter.algorithm.version default depends on runtime environment

2020-08-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32701: Assignee: Apache Spark > mapreduce.fileoutputcommitter.algorithm.version default depends

[jira] [Commented] (SPARK-32701) mapreduce.fileoutputcommitter.algorithm.version default depends on runtime environment

2020-08-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184729#comment-17184729 ] Apache Spark commented on SPARK-32701: -- User 'waleedfateem' has created a pull request for this

[jira] [Assigned] (SPARK-32701) mapreduce.fileoutputcommitter.algorithm.version default depends on runtime environment

2020-08-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32701: Assignee: (was: Apache Spark) > mapreduce.fileoutputcommitter.algorithm.version

[jira] [Commented] (SPARK-32702) Update MiMa plugin

2020-08-25 Thread Denis Pyshev (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184724#comment-17184724 ] Denis Pyshev commented on SPARK-32702: -- This is definitely better (check against 3.0.0):

[jira] [Comment Edited] (SPARK-32702) Update MiMa plugin

2020-08-25 Thread Denis Pyshev (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184716#comment-17184716 ] Denis Pyshev edited comment on SPARK-32702 at 8/25/20, 8:07 PM: >  The

[jira] [Commented] (SPARK-32702) Update MiMa plugin

2020-08-25 Thread Denis Pyshev (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184716#comment-17184716 ] Denis Pyshev commented on SPARK-32702: -- >  The "not analyzing binary compatibility" messages look

[jira] [Updated] (SPARK-32703) Enable dictionary filtering for Parquet vectorized reader

2020-08-25 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-32703: - Description: Parquet vectorized reader still uses the old API for {{filterRowGroups}} and only filters

[jira] [Commented] (SPARK-32702) Update MiMa plugin

2020-08-25 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184708#comment-17184708 ] Sean R. Owen commented on SPARK-32702: -- The "not analyzing binary compatibility" messages look as

[jira] [Updated] (SPARK-32703) Enable dictionary filtering for Parquet vectorized reader

2020-08-25 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-32703: - Summary: Enable dictionary filtering for Parquet vectorized reader (was: Re-enable dictionary

[jira] [Created] (SPARK-32703) Re-enable dictionary filtering for Parquet

2020-08-25 Thread Chao Sun (Jira)
Chao Sun created SPARK-32703: Summary: Re-enable dictionary filtering for Parquet Key: SPARK-32703 URL: https://issues.apache.org/jira/browse/SPARK-32703 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-32702) Update MiMa plugin

2020-08-25 Thread Denis Pyshev (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denis Pyshev updated SPARK-32702: - Attachment: core.txt mllib.txt streaming.txt

[jira] [Updated] (SPARK-32702) Update MiMa plugin

2020-08-25 Thread Denis Pyshev (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denis Pyshev updated SPARK-32702: - Description: As a part of upgrade to SBT 1.x it was found that MiMa (in form of sbt-mima

[jira] [Updated] (SPARK-32702) Update MiMa plugin

2020-08-25 Thread Denis Pyshev (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denis Pyshev updated SPARK-32702: - Description: As a part of upgrade to SBT 1.x it was found that MiMa (in form of sbt-mima

[jira] [Created] (SPARK-32702) Update MiMa plugin

2020-08-25 Thread Denis Pyshev (Jira)
Denis Pyshev created SPARK-32702: Summary: Update MiMa plugin Key: SPARK-32702 URL: https://issues.apache.org/jira/browse/SPARK-32702 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-32701) mapreduce.fileoutputcommitter.algorithm.version default depends on runtime environment

2020-08-25 Thread Waleed Fateem (Jira)
Waleed Fateem created SPARK-32701: - Summary: mapreduce.fileoutputcommitter.algorithm.version default depends on runtime environment Key: SPARK-32701 URL: https://issues.apache.org/jira/browse/SPARK-32701

[jira] [Commented] (SPARK-26164) [SQL] Allow FileFormatWriter to write multiple partitions/buckets without sort

2020-08-25 Thread Cheng Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184623#comment-17184623 ] Cheng Su commented on SPARK-26164: -- Just update - discussed with [~cloud_fan], this Jira is still

[jira] [Commented] (SPARK-32694) Pushdown cast to data sources

2020-08-25 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184614#comment-17184614 ] Chao Sun commented on SPARK-32694: -- Thanks [~rakson] for the pointer! didn't know there were multiple

[jira] [Commented] (SPARK-32699) Add percentage of missingness to df.summary()

2020-08-25 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184176#comment-17184176 ] Sean R. Owen commented on SPARK-32699: -- Pretty easy to compute this if needed, no? > Add

[jira] [Updated] (SPARK-32699) Add percentage of missingness to df.summary()

2020-08-25 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-32699: - Priority: Minor (was: Major) > Add percentage of missingness to df.summary() >

[jira] [Commented] (SPARK-32700) select from table TABLESAMPLE gives wrong resultset.

2020-08-25 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184175#comment-17184175 ] Sean R. Owen commented on SPARK-32700: -- I don't think you're guaranteed to get exactly 50% here.

[jira] [Updated] (SPARK-32659) Fix the data issue of inserted DPP on non-atomic type

2020-08-25 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-32659: Target Version/s: 3.0.1 > Fix the data issue of inserted DPP on non-atomic type >

[jira] [Updated] (SPARK-32659) Fix the data issue of inserted DPP on non-atomic type

2020-08-25 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-32659: Issue Type: Bug (was: Improvement) > Fix the data issue of inserted DPP on non-atomic type >

[jira] [Updated] (SPARK-32659) Fix the data issue of inserted DPP on non-atomic type

2020-08-25 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-32659: Summary: Fix the data issue of inserted DPP on non-atomic type (was: Replace Array with Set in

[jira] [Commented] (SPARK-32037) Rename blacklisting feature to avoid language with racist connotation

2020-08-25 Thread Micah Kornfield (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184144#comment-17184144 ] Micah Kornfield commented on SPARK-32037: - FWIW one argument for using 'denylist'  is it is the

[jira] [Resolved] (SPARK-32614) Support for treating the line as valid record if it starts with \u0000 or null character, or starts with any character mentioned as comment

2020-08-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32614. -- Fix Version/s: 3.1.0 3.0.1 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-32614) Support for treating the line as valid record if it starts with \u0000 or null character, or starts with any character mentioned as comment

2020-08-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32614: Assignee: Sean R. Owen > Support for treating the line as valid record if it starts with

[jira] [Updated] (SPARK-32691) Test org.apache.spark.DistributedSuite failed on arm64 jenkins

2020-08-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32691: -- Environment: ARM64 (was: ARM) > Test org.apache.spark.DistributedSuite failed on arm64

[jira] [Updated] (SPARK-32691) Test org.apache.spark.DistributedSuite failed on arm64 jenkins

2020-08-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32691: -- Environment: ARM > Test org.apache.spark.DistributedSuite failed on arm64 jenkins >

[jira] [Updated] (SPARK-32691) Test org.apache.spark.DistributedSuite failed on arm64 jenkins

2020-08-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32691: -- Component/s: Spark Core > Test org.apache.spark.DistributedSuite failed on arm64 jenkins >

[jira] [Updated] (SPARK-32691) Test org.apache.spark.DistributedSuite failed on arm64 jenkins

2020-08-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32691: -- Issue Type: Bug (was: Test) > Test org.apache.spark.DistributedSuite failed on arm64 jenkins

[jira] [Commented] (SPARK-32691) Test org.apache.spark.DistributedSuite failed on arm64 jenkins

2020-08-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184122#comment-17184122 ] Dongjoon Hyun commented on SPARK-32691: --- Thank you for reporting, [~huangtianhua]. Yes. I suspects

[jira] [Comment Edited] (SPARK-32691) Test org.apache.spark.DistributedSuite failed on arm64 jenkins

2020-08-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184122#comment-17184122 ] Dongjoon Hyun edited comment on SPARK-32691 at 8/25/20, 3:18 PM: - Thank

[jira] [Created] (SPARK-32700) select from table TABLESAMPLE gives wrong resultset.

2020-08-25 Thread Chetan Bhat (Jira)
Chetan Bhat created SPARK-32700: --- Summary: select from table TABLESAMPLE gives wrong resultset. Key: SPARK-32700 URL: https://issues.apache.org/jira/browse/SPARK-32700 Project: Spark Issue

[jira] [Commented] (SPARK-32110) -0.0 vs 0.0 is inconsistent

2020-08-25 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184101#comment-17184101 ] Takeshi Yamamuro commented on SPARK-32110: -- We cannot provide a boolean configuration (turning

[jira] [Created] (SPARK-32699) Add percentage of missingness to df.summary()

2020-08-25 Thread Chengyin Eng (Jira)
Chengyin Eng created SPARK-32699: Summary: Add percentage of missingness to df.summary() Key: SPARK-32699 URL: https://issues.apache.org/jira/browse/SPARK-32699 Project: Spark Issue Type:

[jira] [Updated] (SPARK-31167) Refactor how we track Python test/build dependencies

2020-08-25 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-31167: - Description: Ideally, we should have a single place to track Python development

[jira] [Updated] (SPARK-31167) Refactor how we track Python test/build dependencies

2020-08-25 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-31167: - Description: Ideally, we should have a single place to track Python development

[jira] [Resolved] (SPARK-32664) Getting local shuffle block clutters the executor logs

2020-08-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32664. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29527

[jira] [Assigned] (SPARK-32664) Getting local shuffle block clutters the executor logs

2020-08-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32664: Assignee: Daniel Moore > Getting local shuffle block clutters the executor logs >

[jira] [Commented] (SPARK-32037) Rename blacklisting feature to avoid language with racist connotation

2020-08-25 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184059#comment-17184059 ] Thomas Graves commented on SPARK-32037: --- I started a thread on dev to get feedback: 

[jira] [Commented] (SPARK-32333) Drop references to Master

2020-08-25 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184051#comment-17184051 ] Thomas Graves commented on SPARK-32333: --- I send email to the dev list to get feedback, some other

[jira] [Resolved] (SPARK-32107) Dask faster than Spark with a lot less iterations and better accuracy

2020-08-25 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-32107. -- Resolution: Invalid > Dask faster than Spark with a lot less iterations and better

[jira] [Commented] (SPARK-32107) Dask faster than Spark with a lot less iterations and better accuracy

2020-08-25 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184050#comment-17184050 ] Takeshi Yamamuro commented on SPARK-32107: -- Could you ask the question in the mailing list,

[jira] [Comment Edited] (SPARK-32683) Datetime Pattern F not working as expected

2020-08-25 Thread Daeho Ro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184031#comment-17184031 ] Daeho Ro edited comment on SPARK-32683 at 8/25/20, 1:27 PM: I did not mean

[jira] [Commented] (SPARK-32683) Datetime Pattern F not working as expected

2020-08-25 Thread Daeho Ro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184031#comment-17184031 ] Daeho Ro commented on SPARK-32683: -- I did not mean to change the doc but the source or recover the

[jira] [Assigned] (SPARK-32683) Datetime Pattern F not working as expected

2020-08-25 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-32683: --- Assignee: Kent Yao > Datetime Pattern F not working as expected >

[jira] [Resolved] (SPARK-32683) Datetime Pattern F not working as expected

2020-08-25 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-32683. - Fix Version/s: 3.1.0 3.0.1 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-32698) Do not fall back to default parallelism if the minimum number of coalesced partitions is not set in AQE

2020-08-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32698: Assignee: (was: Apache Spark) > Do not fall back to default parallelism if the

[jira] [Assigned] (SPARK-32698) Do not fall back to default parallelism if the minimum number of coalesced partitions is not set in AQE

2020-08-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32698: Assignee: Apache Spark > Do not fall back to default parallelism if the minimum number

[jira] [Assigned] (SPARK-32698) Do not fall back to default parallelism if the minimum number of coalesced partitions is not set in AQE

2020-08-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32698: Assignee: (was: Apache Spark) > Do not fall back to default parallelism if the

[jira] [Commented] (SPARK-32698) Do not fall back to default parallelism if the minimum number of coalesced partitions is not set in AQE

2020-08-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184009#comment-17184009 ] Apache Spark commented on SPARK-32698: -- User 'manuzhang' has created a pull request for this issue:

[jira] [Commented] (SPARK-32107) Dask faster than Spark with a lot less iterations and better accuracy

2020-08-25 Thread Julian (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184007#comment-17184007 ] Julian commented on SPARK-32107: Dear Spark-Team,   will there be any effort to resolve this issue?

[jira] [Created] (SPARK-32698) Do not fall back to default parallelism if the minimum number of coalesced partitions is not set in AQE

2020-08-25 Thread Manu Zhang (Jira)
Manu Zhang created SPARK-32698: -- Summary: Do not fall back to default parallelism if the minimum number of coalesced partitions is not set in AQE Key: SPARK-32698 URL:

[jira] [Created] (SPARK-32697) Direct Date and timestamp format data insertion fails

2020-08-25 Thread Chetan Bhat (Jira)
Chetan Bhat created SPARK-32697: --- Summary: Direct Date and timestamp format data insertion fails Key: SPARK-32697 URL: https://issues.apache.org/jira/browse/SPARK-32697 Project: Spark Issue

[jira] [Commented] (SPARK-32696) Get columns operation should handle interval column properly

2020-08-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183911#comment-17183911 ] Apache Spark commented on SPARK-32696: -- User 'yaooqinn' has created a pull request for this issue:

[jira] [Commented] (SPARK-32696) Get columns operation should handle interval column properly

2020-08-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183910#comment-17183910 ] Apache Spark commented on SPARK-32696: -- User 'yaooqinn' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32696) Get columns operation should handle interval column properly

2020-08-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32696: Assignee: Apache Spark > Get columns operation should handle interval column properly >

[jira] [Assigned] (SPARK-32696) Get columns operation should handle interval column properly

2020-08-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32696: Assignee: (was: Apache Spark) > Get columns operation should handle interval column

[jira] [Created] (SPARK-32696) Get columns operation should handle interval column properly

2020-08-25 Thread Kent Yao (Jira)
Kent Yao created SPARK-32696: Summary: Get columns operation should handle interval column properly Key: SPARK-32696 URL: https://issues.apache.org/jira/browse/SPARK-32696 Project: Spark Issue

[jira] [Assigned] (SPARK-32683) Datetime Pattern F not working as expected

2020-08-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32683: Assignee: Apache Spark > Datetime Pattern F not working as expected >

[jira] [Assigned] (SPARK-32683) Datetime Pattern F not working as expected

2020-08-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32683: Assignee: (was: Apache Spark) > Datetime Pattern F not working as expected >

[jira] [Commented] (SPARK-32683) Datetime Pattern F not working as expected

2020-08-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183770#comment-17183770 ] Apache Spark commented on SPARK-32683: -- User 'yaooqinn' has created a pull request for this issue:

[jira] [Commented] (SPARK-32683) Datetime Pattern F not working as expected

2020-08-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183769#comment-17183769 ] Apache Spark commented on SPARK-32683: -- User 'yaooqinn' has created a pull request for this issue:

[jira] [Commented] (SPARK-32500) Query and Batch Id not set for Structured Streaming Jobs in case of ForeachBatch in PySpark

2020-08-25 Thread Abhishek Dixit (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183762#comment-17183762 ] Abhishek Dixit commented on SPARK-32500: Thanks [~JinxinTang] and [~hyukjin.kwon] !! > Query

  1   2   >