[jira] [Updated] (SPARK-37513) date +/- interval with only day-time fields returns different data type between Spark3.2 and Spark3.1

2021-11-30 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-37513: --- Summary: date +/- interval with only day-time fields returns different data type between Spark3.2

[jira] [Commented] (SPARK-37210) An error occurred while concurrently writing to different static partitions

2021-11-30 Thread Zhen Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451582#comment-17451582 ] Zhen Wang commented on SPARK-37210: --- I made some adjustments to the test cases to make it more likely

[jira] [Commented] (SPARK-37210) An error occurred while concurrently writing to different static partitions

2021-11-30 Thread Zhen Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451580#comment-17451580 ] Zhen Wang commented on SPARK-37210: --- There seem to be two bugs: # For the insert overwrite fully

[jira] [Updated] (SPARK-37392) Catalyst optimizer very time-consuming and memory-intensive with some "explode(array)"

2021-11-30 Thread Francois MARTIN (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francois MARTIN updated SPARK-37392: Affects Version/s: 3.2.0 > Catalyst optimizer very time-consuming and memory-intensive

[jira] [Assigned] (SPARK-37326) Support TimestampNTZ in CSV data source

2021-11-30 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-37326: Assignee: Ivan Sadikov > Support TimestampNTZ in CSV data source >

[jira] [Resolved] (SPARK-37326) Support TimestampNTZ in CSV data source

2021-11-30 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-37326. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34596

[jira] [Assigned] (SPARK-37480) Configurations in docs/running-on-kubernetes.md are not uptodate

2021-11-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37480: - Assignee: Yikun Jiang > Configurations in docs/running-on-kubernetes.md are not

[jira] [Resolved] (SPARK-37480) Configurations in docs/running-on-kubernetes.md are not uptodate

2021-11-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37480. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34734

[jira] [Commented] (SPARK-37513) Additive expression of date and interval returns different data type between Spark3.2 and Spark3.1

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451542#comment-17451542 ] Apache Spark commented on SPARK-37513: -- User 'beliefer' has created a pull request for this issue:

[jira] [Assigned] (SPARK-37513) Additive expression of date and interval returns different data type between Spark3.2 and Spark3.1

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37513: Assignee: Apache Spark > Additive expression of date and interval returns different data

[jira] [Assigned] (SPARK-37513) Additive expression of date and interval returns different data type between Spark3.2 and Spark3.1

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37513: Assignee: (was: Apache Spark) > Additive expression of date and interval returns

[jira] [Commented] (SPARK-37513) Additive expression of date and interval returns different data type between Spark3.2 and Spark3.1

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451541#comment-17451541 ] Apache Spark commented on SPARK-37513: -- User 'beliefer' has created a pull request for this issue:

[jira] [Updated] (SPARK-37513) Additive expression of date and interval returns different data type between Spark3.2 and Spark3.1

2021-11-30 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-37513: --- Summary: Additive expression of date and interval returns different data type between Spark3.2 and

[jira] [Created] (SPARK-37513) additive expression of date and interval returns different data type between Spark3.2 and Spark3.1

2021-11-30 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-37513: -- Summary: additive expression of date and interval returns different data type between Spark3.2 and Spark3.1 Key: SPARK-37513 URL: https://issues.apache.org/jira/browse/SPARK-37513

[jira] [Commented] (SPARK-37487) CollectMetrics is executed twice if it is followed by a sort

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451530#comment-17451530 ] Apache Spark commented on SPARK-37487: -- User 'sarutak' has created a pull request for this issue:

[jira] [Assigned] (SPARK-37487) CollectMetrics is executed twice if it is followed by a sort

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37487: Assignee: (was: Apache Spark) > CollectMetrics is executed twice if it is followed

[jira] [Assigned] (SPARK-37487) CollectMetrics is executed twice if it is followed by a sort

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37487: Assignee: Apache Spark > CollectMetrics is executed twice if it is followed by a sort >

[jira] [Assigned] (SPARK-37330) Migrate ReplaceTableStatement to v2 command

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37330: Assignee: Apache Spark > Migrate ReplaceTableStatement to v2 command >

[jira] [Commented] (SPARK-37330) Migrate ReplaceTableStatement to v2 command

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451527#comment-17451527 ] Apache Spark commented on SPARK-37330: -- User 'dchvn' has created a pull request for this issue:

[jira] [Assigned] (SPARK-37330) Migrate ReplaceTableStatement to v2 command

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37330: Assignee: (was: Apache Spark) > Migrate ReplaceTableStatement to v2 command >

[jira] [Commented] (SPARK-37330) Migrate ReplaceTableStatement to v2 command

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451529#comment-17451529 ] Apache Spark commented on SPARK-37330: -- User 'dchvn' has created a pull request for this issue:

[jira] [Commented] (SPARK-37389) Check unclosed bracketed comments

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451524#comment-17451524 ] Apache Spark commented on SPARK-37389: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Commented] (SPARK-37389) Check unclosed bracketed comments

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451525#comment-17451525 ] Apache Spark commented on SPARK-37389: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Created] (SPARK-37512) Support TimedeltaIndex creation given a timedelta Series/Index

2021-11-30 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-37512: Summary: Support TimedeltaIndex creation given a timedelta Series/Index Key: SPARK-37512 URL: https://issues.apache.org/jira/browse/SPARK-37512 Project: Spark

[jira] [Assigned] (SPARK-37376) Introduce a new DataSource V2 interface HasPartitionKey

2021-11-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37376: - Assignee: Chao Sun > Introduce a new DataSource V2 interface HasPartitionKey >

[jira] [Resolved] (SPARK-37376) Introduce a new DataSource V2 interface HasPartitionKey

2021-11-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37376. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34656

[jira] [Resolved] (SPARK-37490) Show hint if analyzer fails due to ANSI type coercion

2021-11-30 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-37490. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34747

[jira] [Commented] (SPARK-37476) udaf doesnt work with nullable (or option of) case class result

2021-11-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451496#comment-17451496 ] Hyukjin Kwon commented on SPARK-37476: -- can you try {{java.lang.Double}} that can legitimately be

[jira] [Assigned] (SPARK-37504) pyspark should not pass all options to session states.

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37504: Assignee: (was: Apache Spark) > pyspark should not pass all options to session

[jira] [Commented] (SPARK-37504) pyspark should not pass all options to session states.

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451483#comment-17451483 ] Apache Spark commented on SPARK-37504: -- User 'AngersZh' has created a pull request for this

[jira] [Assigned] (SPARK-37504) pyspark should not pass all options to session states.

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37504: Assignee: Apache Spark > pyspark should not pass all options to session states. >

[jira] [Commented] (SPARK-37487) CollectMetrics is executed twice if it is followed by a sort

2021-11-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451482#comment-17451482 ] Hyukjin Kwon commented on SPARK-37487: -- cc [~beliefer] FYI > CollectMetrics is executed twice if

[jira] [Commented] (SPARK-37498) test_reuse_worker_of_parallelize_range is flaky

2021-11-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451481#comment-17451481 ] Hyukjin Kwon commented on SPARK-37498: -- maybe we should add eventually

[jira] [Assigned] (SPARK-37503) Improve SparkSession startup issue

2021-11-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-37503: Assignee: angerszhu > Improve SparkSession startup issue >

[jira] [Updated] (SPARK-37497) Promote ExecutorPods[PollingSnapshot|WatchSnapshot]Source to DeveloperApi

2021-11-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37497: -- Issue Type: Task (was: Improvement) > Promote

[jira] [Updated] (SPARK-37511) Introduce TimedeltaIndex to pandas API on Spark

2021-11-30 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-37511: - Description: Introduce TimedeltaIndex to pandas API on Spark. Properties, functions, and basic

[jira] [Resolved] (SPARK-37497) Promote ExecutorPods[PollingSnapshot|WatchSnapshot]Source to DeveloperApi

2021-11-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37497. --- Fix Version/s: 3.3.0 3.2.1 3.1.3 Resolution:

[jira] [Assigned] (SPARK-37497) Promote ExecutorPods[PollingSnapshot|WatchSnapshot]Source to DeveloperApi

2021-11-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37497: - Assignee: Dongjoon Hyun > Promote ExecutorPods[PollingSnapshot|WatchSnapshot]Source to

[jira] [Assigned] (SPARK-37511) Introduce TimedeltaIndex to pandas API on Spark

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37511: Assignee: Apache Spark > Introduce TimedeltaIndex to pandas API on Spark >

[jira] [Commented] (SPARK-37511) Introduce TimedeltaIndex to pandas API on Spark

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451472#comment-17451472 ] Apache Spark commented on SPARK-37511: -- User 'xinrong-databricks' has created a pull request for

[jira] [Assigned] (SPARK-37511) Introduce TimedeltaIndex to pandas API on Spark

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37511: Assignee: (was: Apache Spark) > Introduce TimedeltaIndex to pandas API on Spark >

[jira] [Created] (SPARK-37511) Introduce TimedeltaIndex to pandas API on Spark

2021-11-30 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-37511: Summary: Introduce TimedeltaIndex to pandas API on Spark Key: SPARK-37511 URL: https://issues.apache.org/jira/browse/SPARK-37511 Project: Spark Issue Type:

[jira] [Created] (SPARK-37510) Support TimedeltaIndex in pandas API on Spark

2021-11-30 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-37510: Summary: Support TimedeltaIndex in pandas API on Spark Key: SPARK-37510 URL: https://issues.apache.org/jira/browse/SPARK-37510 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-37509) Improve Fallback Storage upload speed by avoiding S3 rate limiter

2021-11-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37509. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34762

[jira] [Assigned] (SPARK-37509) Improve Fallback Storage upload speed by avoiding S3 rate limiter

2021-11-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37509: - Assignee: Dongjoon Hyun > Improve Fallback Storage upload speed by avoiding S3 rate

[jira] [Resolved] (SPARK-37505) mesos module is missing log4j.properties file for UT

2021-11-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37505. --- Fix Version/s: 3.3.0 3.2.1 3.1.3 Resolution:

[jira] [Assigned] (SPARK-37505) mesos module is missing log4j.properties file for UT

2021-11-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37505: - Assignee: Yang Jie > mesos module is missing log4j.properties file for UT >

[jira] [Commented] (SPARK-37487) CollectMetrics is executed twice if it is followed by a sort

2021-11-30 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451376#comment-17451376 ] Kousuke Saruta commented on SPARK-37487: [~tanelk] Thank you for pinging me. I think a sampling

[jira] [Commented] (SPARK-37509) Improve Fallback Storage upload speed by avoiding S3 rate limiter

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451308#comment-17451308 ] Apache Spark commented on SPARK-37509: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-37509) Improve Fallback Storage upload speed by avoiding S3 rate limiter

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37509: Assignee: (was: Apache Spark) > Improve Fallback Storage upload speed by avoiding S3

[jira] [Assigned] (SPARK-37509) Improve Fallback Storage upload speed by avoiding S3 rate limiter

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37509: Assignee: Apache Spark > Improve Fallback Storage upload speed by avoiding S3 rate

[jira] [Created] (SPARK-37509) Improve Fallback Storage upload speed by avoiding S3 rate limiter

2021-11-30 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-37509: - Summary: Improve Fallback Storage upload speed by avoiding S3 rate limiter Key: SPARK-37509 URL: https://issues.apache.org/jira/browse/SPARK-37509 Project: Spark

[jira] [Comment Edited] (SPARK-26589) proper `median` method for spark dataframe

2021-11-30 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451283#comment-17451283 ] Nicholas Chammas edited comment on SPARK-26589 at 11/30/21, 6:17 PM: -

[jira] [Commented] (SPARK-26589) proper `median` method for spark dataframe

2021-11-30 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451283#comment-17451283 ] Nicholas Chammas commented on SPARK-26589: -- I'm going to try to implement this using the

[jira] [Updated] (SPARK-12185) Add Histogram support to Spark SQL/DataFrames

2021-11-30 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-12185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-12185: - Labels: (was: bulk-closed) > Add Histogram support to Spark SQL/DataFrames >

[jira] [Reopened] (SPARK-12185) Add Histogram support to Spark SQL/DataFrames

2021-11-30 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-12185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas reopened SPARK-12185: -- Reopening this because I think it's a valid improvement that mirrors the existing

[jira] [Commented] (SPARK-37508) Add CONTAINS() function

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451203#comment-17451203 ] Apache Spark commented on SPARK-37508: -- User 'AngersZh' has created a pull request for this

[jira] [Assigned] (SPARK-37508) Add CONTAINS() function

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37508: Assignee: Apache Spark > Add CONTAINS() function > --- > >

[jira] [Assigned] (SPARK-37508) Add CONTAINS() function

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37508: Assignee: (was: Apache Spark) > Add CONTAINS() function > --- >

[jira] [Commented] (SPARK-37508) Add CONTAINS() function

2021-11-30 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451183#comment-17451183 ] angerszhu commented on SPARK-37508: --- Raise a pr soon > Add CONTAINS() function >

[jira] [Updated] (SPARK-37461) yarn-client mode client's appid value is null

2021-11-30 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-37461: -- Description: In yarn-client mode, *Client.appId* variable is not assigned, it is always `null`, in

[jira] [Updated] (SPARK-37461) yarn-client mode client's appid value is null

2021-11-30 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-37461: -- Description: In yarn-client mode, *Client.appId* variable is not assigned, it is always {*}null{*},

[jira] [Updated] (SPARK-37461) yarn-client mode client's appid value is null

2021-11-30 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-37461: -- Description: In yarn-client mode, *Client.appId* variable is not assigned, it is always {*}null{*},

[jira] [Commented] (SPARK-37461) yarn-client mode client's appid value is null

2021-11-30 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451172#comment-17451172 ] Thomas Graves commented on SPARK-37461: --- [~angerszhuuu] please add a description to this issue. >

[jira] [Comment Edited] (SPARK-10925) Exception when joining DataFrames

2021-11-30 Thread Joao Miguel Pinto (Jira)
[ https://issues.apache.org/jira/browse/SPARK-10925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451155#comment-17451155 ] Joao Miguel Pinto edited comment on SPARK-10925 at 11/30/21, 2:34 PM:

[jira] [Comment Edited] (SPARK-10925) Exception when joining DataFrames

2021-11-30 Thread Joao Miguel Pinto (Jira)
[ https://issues.apache.org/jira/browse/SPARK-10925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451155#comment-17451155 ] Joao Miguel Pinto edited comment on SPARK-10925 at 11/30/21, 2:20 PM:

[jira] [Commented] (SPARK-10925) Exception when joining DataFrames

2021-11-30 Thread Joao Miguel Pinto (Jira)
[ https://issues.apache.org/jira/browse/SPARK-10925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451155#comment-17451155 ] Joao Miguel Pinto commented on SPARK-10925: --- Hi guys. I had the same problem and the only way

[jira] [Commented] (SPARK-37508) Add CONTAINS() function

2021-11-30 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451149#comment-17451149 ] Max Gekk commented on SPARK-37508: -- [~angerszhuuu] [~xiaopenglei] @jiaan.geng If you would like to work

[jira] [Comment Edited] (SPARK-37508) Add CONTAINS() function

2021-11-30 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451149#comment-17451149 ] Max Gekk edited comment on SPARK-37508 at 11/30/21, 2:09 PM: - [~angerszhuuu]

[jira] [Updated] (SPARK-37508) Add CONTAINS() function

2021-11-30 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-37508: - Description: {{contains()}} is a common convenient function supported by a number of database systems:

[jira] [Created] (SPARK-37508) Add CONTAINS() function

2021-11-30 Thread Max Gekk (Jira)
Max Gekk created SPARK-37508: Summary: Add CONTAINS() function Key: SPARK-37508 URL: https://issues.apache.org/jira/browse/SPARK-37508 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-37507) Add the TO_BINARY() function

2021-11-30 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451138#comment-17451138 ] Max Gekk commented on SPARK-37507: -- [~beliefer] Would you like to work on this? If so, please, leave a

[jira] [Created] (SPARK-37507) Add the TO_BINARY() function

2021-11-30 Thread Max Gekk (Jira)
Max Gekk created SPARK-37507: Summary: Add the TO_BINARY() function Key: SPARK-37507 URL: https://issues.apache.org/jira/browse/SPARK-37507 Project: Spark Issue Type: New Feature

[jira] [Assigned] (SPARK-37506) Change the never changed 'var' to 'val'

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37506: Assignee: Apache Spark > Change the never changed 'var' to 'val' >

[jira] [Assigned] (SPARK-37506) Change the never changed 'var' to 'val'

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37506: Assignee: (was: Apache Spark) > Change the never changed 'var' to 'val' >

[jira] [Commented] (SPARK-37506) Change the never changed 'var' to 'val'

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451086#comment-17451086 ] Apache Spark commented on SPARK-37506: -- User 'LuciferYang' has created a pull request for this

[jira] [Created] (SPARK-37506) Change the never changed 'var' to 'val'

2021-11-30 Thread Yang Jie (Jira)
Yang Jie created SPARK-37506: Summary: Change the never changed 'var' to 'val' Key: SPARK-37506 URL: https://issues.apache.org/jira/browse/SPARK-37506 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-37505) mesos module is missing log4j.properties file for UT

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37505: Assignee: (was: Apache Spark) > mesos module is missing log4j.properties file for UT

[jira] [Assigned] (SPARK-37505) mesos module is missing log4j.properties file for UT

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37505: Assignee: Apache Spark > mesos module is missing log4j.properties file for UT >

[jira] [Commented] (SPARK-37505) mesos module is missing log4j.properties file for UT

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451077#comment-17451077 ] Apache Spark commented on SPARK-37505: -- User 'LuciferYang' has created a pull request for this

[jira] [Created] (SPARK-37505) mesos module is missing log4j.properties file for UT

2021-11-30 Thread Yang Jie (Jira)
Yang Jie created SPARK-37505: Summary: mesos module is missing log4j.properties file for UT Key: SPARK-37505 URL: https://issues.apache.org/jira/browse/SPARK-37505 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-37501) CREATE/REPLACE TABLE should qualify location for v2 command

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37501: Assignee: Apache Spark > CREATE/REPLACE TABLE should qualify location for v2 command >

[jira] [Assigned] (SPARK-37501) CREATE/REPLACE TABLE should qualify location for v2 command

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37501: Assignee: (was: Apache Spark) > CREATE/REPLACE TABLE should qualify location for v2

[jira] [Commented] (SPARK-37501) CREATE/REPLACE TABLE should qualify location for v2 command

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451057#comment-17451057 ] Apache Spark commented on SPARK-37501: -- User 'Peng-Lei' has created a pull request for this issue:

[jira] [Commented] (SPARK-37054) Porting "pandas API on Spark: Internals" to PySpark docs.

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451005#comment-17451005 ] Apache Spark commented on SPARK-37054: -- User 'AngersZh' has created a pull request for this

[jira] [Updated] (SPARK-37504) pyspark should not pass all options to session states.

2021-11-30 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-37504: -- Description: in current session.py have Such code {code} for key, value in self._options.items():

[jira] [Created] (SPARK-37504) pyspark should not pass all options to session states.

2021-11-30 Thread angerszhu (Jira)
angerszhu created SPARK-37504: - Summary: pyspark should not pass all options to session states. Key: SPARK-37504 URL: https://issues.apache.org/jira/browse/SPARK-37504 Project: Spark Issue Type:

[jira] [Updated] (SPARK-34568) enableHiveSupport should ignore if SparkContext is created

2021-11-30 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-34568: -- Parent: SPARK-37503 Issue Type: Sub-task (was: Bug) > enableHiveSupport should ignore if

[jira] [Resolved] (SPARK-37346) Link migration guide for structured stream.

2021-11-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37346. -- Assignee: angerszhu Resolution: Fixed > Link migration guide for structured stream. >

[jira] [Updated] (SPARK-37346) Link migration guide for structured stream.

2021-11-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37346: - Fix Version/s: 3.1.3 3.2.1 > Link migration guide for structured stream. >

[jira] [Updated] (SPARK-37291) PySpark init SparkSession should copy conf to sharedState

2021-11-30 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-37291: -- Parent: SPARK-37503 Issue Type: Sub-task (was: Bug) > PySpark init SparkSession should copy

[jira] [Created] (SPARK-37503) Improve SparkSession startup issue

2021-11-30 Thread angerszhu (Jira)
angerszhu created SPARK-37503: - Summary: Improve SparkSession startup issue Key: SPARK-37503 URL: https://issues.apache.org/jira/browse/SPARK-37503 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-37502) Support cast aware output partitioning and required if it can up cast

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37502: Assignee: Apache Spark > Support cast aware output partitioning and required if it can

[jira] [Assigned] (SPARK-37502) Support cast aware output partitioning and required if it can up cast

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37502: Assignee: (was: Apache Spark) > Support cast aware output partitioning and required

[jira] [Commented] (SPARK-37502) Support cast aware output partitioning and required if it can up cast

2021-11-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450966#comment-17450966 ] Apache Spark commented on SPARK-37502: -- User 'ulysses-you' has created a pull request for this

[jira] [Updated] (SPARK-37502) Support cast aware output partitioning and required if it can up cast

2021-11-30 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-37502: -- Description: If a `Cast` is up cast then it should be without any truncating or precision lose or

[jira] [Updated] (SPARK-37501) CREATE/REPLACE TABLE should qualify location for v2 command

2021-11-30 Thread PengLei (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PengLei updated SPARK-37501: Summary: CREATE/REPLACE TABLE should qualify location for v2 command (was: CREATE TABLE should qualify

[jira] [Created] (SPARK-37502) Support cast aware output partitioning and required if it can up cast

2021-11-30 Thread XiDuo You (Jira)
XiDuo You created SPARK-37502: - Summary: Support cast aware output partitioning and required if it can up cast Key: SPARK-37502 URL: https://issues.apache.org/jira/browse/SPARK-37502 Project: Spark

[jira] [Comment Edited] (SPARK-37381) Unify v1 and v2 SHOW CREATE TABLE tests

2021-11-30 Thread dch nguyen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450947#comment-17450947 ] dch nguyen edited comment on SPARK-37381 at 11/30/21, 8:33 AM: ---

[jira] [Commented] (SPARK-37381) Unify v1 and v2 SHOW CREATE TABLE tests

2021-11-30 Thread dch nguyen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450947#comment-17450947 ] dch nguyen commented on SPARK-37381: [~xiaopenglei] it's ok. Thank you. > Unify v1 and v2 SHOW

  1   2   >