[jira] [Resolved] (SPARK-30020) ownerName and ownerType support as properties to tables

[ https://issues.apache.org/jira/browse/SPARK-30020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-30020. -- Fix Version/s: 3.0.0 Target Version/s: 3.0.0 Resolution: Not A Problem > ownerName

[jira] [Updated] (SPARK-30516) statistic estimation of FileScan should take partitionFilters and partition number into account

[ https://issues.apache.org/jira/browse/SPARK-30516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hu Fuwang updated SPARK-30516: -- Summary: statistic estimation of FileScan should take partitionFilters and partition number into

[jira] [Commented] (SPARK-22184) GraphX fails in case of insufficient memory and checkpoints enabled

[ https://issues.apache.org/jira/browse/SPARK-22184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015693#comment-17015693 ] Takeshi Yamamuro commented on SPARK-22184: -- See

[jira] [Resolved] (SPARK-22184) GraphX fails in case of insufficient memory and checkpoints enabled

[ https://issues.apache.org/jira/browse/SPARK-22184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-22184. -- Resolution: Won't Fix > GraphX fails in case of insufficient memory and checkpoints

[jira] [Resolved] (SPARK-30505) Deprecate Avro option `ignoreExtension` in a doc

[ https://issues.apache.org/jira/browse/SPARK-30505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30505. -- Fix Version/s: 3.0.0 Assignee: Maxim Gekk Resolution: Fixed Fixed in 

[jira] [Commented] (SPARK-22231) Support of map, filter, withColumn, dropColumn in nested list of structures

[ https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015705#comment-17015705 ] Reynold Xin commented on SPARK-22231: - Hey sorry. Been pretty busy. I will take a look this week. >

[jira] [Created] (SPARK-30517) Support SHOW TABLES EXTENDED

Ajith S created SPARK-30517: --- Summary: Support SHOW TABLES EXTENDED Key: SPARK-30517 URL: https://issues.apache.org/jira/browse/SPARK-30517 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-30515) Refactor SimplifyBinaryComparison to reduce time complexity

Gengliang Wang created SPARK-30515: -- Summary: Refactor SimplifyBinaryComparison to reduce time complexity Key: SPARK-30515 URL: https://issues.apache.org/jira/browse/SPARK-30515 Project: Spark

[jira] [Updated] (SPARK-30516) FileScan.estimateStatistics does not take partitionFilters and partition number into account

[ https://issues.apache.org/jira/browse/SPARK-30516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hu Fuwang updated SPARK-30516: -- Description: Currently, FileScan.estimateStatistics does not take partitionFilters and partition

[jira] [Created] (SPARK-30516) FileScan.estimateStatistics does not take partitionFilters and partition number into account

Hu Fuwang created SPARK-30516: - Summary: FileScan.estimateStatistics does not take partitionFilters and partition number into account Key: SPARK-30516 URL: https://issues.apache.org/jira/browse/SPARK-30516

[jira] [Updated] (SPARK-30516) FileScan.estimateStatistics does not take partitionFilters and partition number into account

[ https://issues.apache.org/jira/browse/SPARK-30516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hu Fuwang updated SPARK-30516: -- Description: Currently, FileScan.estimateStatistics will not take partitionFilters into account,

[jira] [Updated] (SPARK-30515) Refactor SimplifyBinaryComparison to reduce the time complexity

[ https://issues.apache.org/jira/browse/SPARK-30515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-30515: --- Summary: Refactor SimplifyBinaryComparison to reduce the time complexity (was: Refactor

[jira] [Assigned] (SPARK-9478) Add sample weights to Random Forest

[ https://issues.apache.org/jira/browse/SPARK-9478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-9478: --- Assignee: zhengruifeng > Add sample weights to Random Forest >

[jira] [Resolved] (SPARK-30423) Deprecate UserDefinedAggregateFunction

[ https://issues.apache.org/jira/browse/SPARK-30423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30423. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27193

[jira] [Resolved] (SPARK-9478) Add sample weights to Random Forest

[ https://issues.apache.org/jira/browse/SPARK-9478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-9478. - Fix Version/s: 3.0.0 Resolution: Fixed > Add sample weights to Random Forest >

[jira] [Reopened] (SPARK-9478) Add sample weights to Random Forest

[ https://issues.apache.org/jira/browse/SPARK-9478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reopened SPARK-9478: - > Add sample weights to Random Forest > --- > > Key:

[jira] [Commented] (SPARK-30424) Change ExpressionEncoder toRow method to return UnsafeRow

[ https://issues.apache.org/jira/browse/SPARK-30424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015183#comment-17015183 ] Erik Erlandson commented on SPARK-30424: The main place this change causes a compile fail on is

[jira] [Commented] (SPARK-30495) How to disable 'spark.security.credentials.${service}.enabled' in Structured streaming while connecting to a kafka cluster

[ https://issues.apache.org/jira/browse/SPARK-30495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015335#comment-17015335 ] Jungtaek Lim commented on SPARK-30495: -- Adjusted priority as it's a regression. May need higher

[jira] [Assigned] (SPARK-27142) Provide REST API for SQL level information

[ https://issues.apache.org/jira/browse/SPARK-27142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin reassigned SPARK-27142: -- Assignee: Ajith S > Provide REST API for SQL level information >

[jira] [Resolved] (SPARK-27142) Provide REST API for SQL level information

[ https://issues.apache.org/jira/browse/SPARK-27142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin resolved SPARK-27142. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Commented] (SPARK-30488) Deadlock between block-manager-slave-async-thread-pool and spark context cleaner

[ https://issues.apache.org/jira/browse/SPARK-30488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015308#comment-17015308 ] Rohit Agrawal commented on SPARK-30488: --- [~ajithshetty] We use the following to create spark

[jira] [Updated] (SPARK-30495) How to disable 'spark.security.credentials.${service}.enabled' in Structured streaming while connecting to a kafka cluster

[ https://issues.apache.org/jira/browse/SPARK-30495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-30495: - Priority: Major (was: Minor) > How to disable 'spark.security.credentials.${service}.enabled'

[jira] [Created] (SPARK-30510) Document spark.sql.sources.partitionOverwriteMode

Nicholas Chammas created SPARK-30510: Summary: Document spark.sql.sources.partitionOverwriteMode Key: SPARK-30510 URL: https://issues.apache.org/jira/browse/SPARK-30510 Project: Spark

[jira] [Created] (SPARK-30511) Spark marks ended speculative tasks as pending leads to holding idle executors

Zebing Lin created SPARK-30511: -- Summary: Spark marks ended speculative tasks as pending leads to holding idle executors Key: SPARK-30511 URL: https://issues.apache.org/jira/browse/SPARK-30511 Project:

[jira] [Updated] (SPARK-30511) Spark marks ended speculative tasks as pending leads to holding idle executors

[ https://issues.apache.org/jira/browse/SPARK-30511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zebing Lin updated SPARK-30511: --- External issue ID: (was: SPARK-2840) > Spark marks ended speculative tasks as pending leads to

[jira] [Updated] (SPARK-30511) Spark marks ended speculative tasks as pending leads to holding idle executors

[ https://issues.apache.org/jira/browse/SPARK-30511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zebing Lin updated SPARK-30511: --- External issue ID: SPARK-2840 > Spark marks ended speculative tasks as pending leads to holding

[jira] [Updated] (SPARK-30511) Spark marks ended speculative tasks as pending leads to holding idle executors

[ https://issues.apache.org/jira/browse/SPARK-30511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zebing Lin updated SPARK-30511: --- Description: *TL;DR* When speculative tasks finished/failed/got killed, they are still considered

[jira] [Created] (SPARK-30512) Use a dedicated boss event group loop in the netty pipeline for external shuffle service

Chandni Singh created SPARK-30512: - Summary: Use a dedicated boss event group loop in the netty pipeline for external shuffle service Key: SPARK-30512 URL: https://issues.apache.org/jira/browse/SPARK-30512

[jira] [Updated] (SPARK-30512) Use a dedicated boss event group loop in the netty pipeline for external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-30512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated SPARK-30512: -- Description: We have been seeing a large number of SASL authentication (RPC requests) timing

[jira] [Commented] (SPARK-30512) Use a dedicated boss event group loop in the netty pipeline for external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-30512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015362#comment-17015362 ] Chandni Singh commented on SPARK-30512: --- Please assign the issue to me so I can open up a PR. >

[jira] [Resolved] (SPARK-30509) Deprecation log warning is not printed in Avro schema inferring

[ https://issues.apache.org/jira/browse/SPARK-30509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30509. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27200

[jira] [Assigned] (SPARK-30509) Deprecation log warning is not printed in Avro schema inferring

[ https://issues.apache.org/jira/browse/SPARK-30509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-30509: - Assignee: Maxim Gekk > Deprecation log warning is not printed in Avro schema inferring

[jira] [Updated] (SPARK-29721) Spark SQL reads unnecessary nested fields from Parquet after using explode

[ https://issues.apache.org/jira/browse/SPARK-29721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29721: -- Affects Version/s: 3.0.0 > Spark SQL reads unnecessary nested fields from Parquet after using

[jira] [Updated] (SPARK-29721) Spark SQL reads unnecessary nested fields from Parquet after using explode

[ https://issues.apache.org/jira/browse/SPARK-29721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29721: -- Affects Version/s: 2.4.0 2.4.1 2.4.2

[jira] [Commented] (SPARK-30510) Document spark.sql.sources.partitionOverwriteMode

[ https://issues.apache.org/jira/browse/SPARK-30510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015473#comment-17015473 ] Nicholas Chammas commented on SPARK-30510: -- [~hyukjin.kwon] I think I'm missing something here

[jira] [Commented] (SPARK-30510) Document spark.sql.sources.partitionOverwriteMode

[ https://issues.apache.org/jira/browse/SPARK-30510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015501#comment-17015501 ] Hyukjin Kwon commented on SPARK-30510: -- I think currently only some of important configurations are

[jira] [Created] (SPARK-30513) Question about spark on k8s

Jackey Lee created SPARK-30513: -- Summary: Question about spark on k8s Key: SPARK-30513 URL: https://issues.apache.org/jira/browse/SPARK-30513 Project: Spark Issue Type: Question

[jira] [Created] (SPARK-30514) add ENV_PYSPARK_MAJOR_PYTHON_VERSION support for JavaMainAppResource

Jackey Lee created SPARK-30514: -- Summary: add ENV_PYSPARK_MAJOR_PYTHON_VERSION support for JavaMainAppResource Key: SPARK-30514 URL: https://issues.apache.org/jira/browse/SPARK-30514 Project: Spark

[jira] [Resolved] (SPARK-22783) event log directory(spark-history) filled by large .inprogress files for spark streaming applications

[ https://issues.apache.org/jira/browse/SPARK-22783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-22783. -- Resolution: Duplicate I'll mark this as "duplicated" as SPARK-28594 is making over half of

[jira] [Resolved] (SPARK-30292) Throw Exception when invalid string is cast to decimal in ANSI mode

[ https://issues.apache.org/jira/browse/SPARK-30292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30292. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26933

[jira] [Assigned] (SPARK-30292) Throw Exception when invalid string is cast to decimal in ANSI mode

[ https://issues.apache.org/jira/browse/SPARK-30292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-30292: --- Assignee: Rakesh Raushan > Throw Exception when invalid string is cast to decimal in ANSI

[jira] [Assigned] (SPARK-30498) Fix some ml parity issues between python and scala

[ https://issues.apache.org/jira/browse/SPARK-30498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng reassigned SPARK-30498: Assignee: Huaxin Gao > Fix some ml parity issues between python and scala >

[jira] [Resolved] (SPARK-30498) Fix some ml parity issues between python and scala

[ https://issues.apache.org/jira/browse/SPARK-30498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng resolved SPARK-30498. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27196

[jira] [Commented] (SPARK-28242) DataStreamer keeps logging errors even after fixing writeStream output sink

[ https://issues.apache.org/jira/browse/SPARK-28242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014919#comment-17014919 ] Hyokun Park commented on SPARK-28242: - Hi [~mcanes] In my case, I resolved the problem by adding a

[jira] [Created] (SPARK-30508) Add DataFrameReader.executeCommand API for external datasource

wuyi created SPARK-30508: Summary: Add DataFrameReader.executeCommand API for external datasource Key: SPARK-30508 URL: https://issues.apache.org/jira/browse/SPARK-30508 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-30325) markPartitionCompleted cause task status inconsistent

[ https://issues.apache.org/jira/browse/SPARK-30325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30325. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26975

[jira] [Assigned] (SPARK-30325) markPartitionCompleted cause task status inconsistent

[ https://issues.apache.org/jira/browse/SPARK-30325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-30325: --- Assignee: haiyangyu > markPartitionCompleted cause task status inconsistent >

[jira] [Assigned] (SPARK-29544) Optimize skewed join at runtime with new Adaptive Execution

[ https://issues.apache.org/jira/browse/SPARK-29544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-29544: --- Assignee: Ke Jia > Optimize skewed join at runtime with new Adaptive Execution >

[jira] [Resolved] (SPARK-29544) Optimize skewed join at runtime with new Adaptive Execution

[ https://issues.apache.org/jira/browse/SPARK-29544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-29544. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26434

[jira] [Created] (SPARK-30509) Deprecation log warning is not printed in Avro schema inferring

Maxim Gekk created SPARK-30509: -- Summary: Deprecation log warning is not printed in Avro schema inferring Key: SPARK-30509 URL: https://issues.apache.org/jira/browse/SPARK-30509 Project: Spark

[jira] [Commented] (SPARK-30295) Remove Hive dependencies from SparkSQLCLI

[ https://issues.apache.org/jira/browse/SPARK-30295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015028#comment-17015028 ] Javier Fuentes commented on SPARK-30295:  Yes, this purely to try to remove more hive