[jira] [Commented] (SPARK-32195) Standardize warning types and messages

2020-10-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17220496#comment-17220496 ] Hyukjin Kwon commented on SPARK-32195: -- It doesn't have to be assigned. Once a PR is open, that

[jira] [Commented] (SPARK-32195) Standardize warning types and messages

2020-10-25 Thread Kevin Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17220492#comment-17220492 ] Kevin Su commented on SPARK-32195: -- Hi, [~hyukjin.kwon]. I'm Kevin, could you assign this issue to me.

[jira] [Created] (SPARK-33241) Dynamic pruning for data column

2020-10-25 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-33241: --- Summary: Dynamic pruning for data column Key: SPARK-33241 URL: https://issues.apache.org/jira/browse/SPARK-33241 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-33240) Fail fast when fails to instantiate configured v2 session catalog

2020-10-25 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17220487#comment-17220487 ] Jungtaek Lim commented on SPARK-33240: -- Will work on it. > Fail fast when fails to instantiate

[jira] [Created] (SPARK-33240) Fail fast when fails to instantiate configured v2 session catalog

2020-10-25 Thread Jungtaek Lim (Jira)
Jungtaek Lim created SPARK-33240: Summary: Fail fast when fails to instantiate configured v2 session catalog Key: SPARK-33240 URL: https://issues.apache.org/jira/browse/SPARK-33240 Project: Spark

[jira] [Commented] (SPARK-32194) Standardize exceptions in PySpark

2020-10-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17220477#comment-17220477 ] Hyukjin Kwon commented on SPARK-32194: -- There should be some kind of simple design for that. It

[jira] [Commented] (SPARK-32195) Standardize warning types and messages

2020-10-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17220478#comment-17220478 ] Hyukjin Kwon commented on SPARK-32195: -- We'll have to follow

[jira] [Assigned] (SPARK-32862) Left semi stream-stream join

2020-10-25 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-32862: Assignee: Cheng Su > Left semi stream-stream join > > >

[jira] [Resolved] (SPARK-32862) Left semi stream-stream join

2020-10-25 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-32862. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30076

[jira] [Commented] (SPARK-32194) Standardize exceptions in PySpark

2020-10-25 Thread RISHAV DUTTA (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17220469#comment-17220469 ] RISHAV DUTTA commented on SPARK-32194: -- I would like to work on this.Please provide a guide on how

[jira] [Commented] (SPARK-32195) Standardize warning types and messages

2020-10-25 Thread RISHAV DUTTA (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17220468#comment-17220468 ] RISHAV DUTTA commented on SPARK-32195: -- Hi is there a guide on how to standardise the exceptions?

[jira] [Updated] (SPARK-32390) TRANSFORM with hive serde support CalendarIntervalType and UserDefinedType

2020-10-25 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-32390: -- Description: convert as string to support. UTD 

[jira] [Reopened] (SPARK-32390) TRANSFORM with hive serde support CalendarIntervalType and UserDefinedType

2020-10-25 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu reopened SPARK-32390: --- need to implement > TRANSFORM with hive serde support CalendarIntervalType and UserDefinedType >

[jira] [Commented] (SPARK-33239) Use pre-built image at GitHub Action SparkR job

2020-10-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17220458#comment-17220458 ] Apache Spark commented on SPARK-33239: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Commented] (SPARK-33239) Use pre-built image at GitHub Action SparkR job

2020-10-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17220459#comment-17220459 ] Apache Spark commented on SPARK-33239: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-33239) Use pre-built image at GitHub Action SparkR job

2020-10-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33239: Assignee: Dongjoon Hyun (was: Apache Spark) > Use pre-built image at GitHub Action

[jira] [Assigned] (SPARK-33239) Use pre-built image at GitHub Action SparkR job

2020-10-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33239: Assignee: Apache Spark (was: Dongjoon Hyun) > Use pre-built image at GitHub Action

[jira] [Resolved] (SPARK-32390) TRANSFORM with hive serde support CalendarIntervalType and UserDefinedType

2020-10-25 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu resolved SPARK-32390. --- Resolution: Fixed > TRANSFORM with hive serde support CalendarIntervalType and UserDefinedType >

[jira] [Created] (SPARK-33239) Use pre-built image at GitHub Action SparkR job

2020-10-25 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-33239: - Summary: Use pre-built image at GitHub Action SparkR job Key: SPARK-33239 URL: https://issues.apache.org/jira/browse/SPARK-33239 Project: Spark Issue

[jira] [Updated] (SPARK-32390) TRANSFORM with hive serde support CalendarIntervalType and UserDefinedType

2020-10-25 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-32390: -- Description: convert as string to support. UTD 

[jira] [Updated] (SPARK-31937) Support processing array and map type using spark noserde mode

2020-10-25 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-31937: -- Description: Currently, It is not supported to use script(e.g. python) to process array type or map

[jira] [Created] (SPARK-33238) Add a configuration to control the legacy behavior of whether need to pad null value when value size less then schema size

2020-10-25 Thread angerszhu (Jira)
angerszhu created SPARK-33238: - Summary: Add a configuration to control the legacy behavior of whether need to pad null value when value size less then schema size Key: SPARK-33238 URL:

[jira] [Assigned] (SPARK-32388) TRANSFORM when schema less should keep same with hive

2020-10-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32388: Assignee: Apache Spark (was: angerszhu) > TRANSFORM when schema less should keep same

[jira] [Assigned] (SPARK-32388) TRANSFORM when schema less should keep same with hive

2020-10-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32388: Assignee: angerszhu (was: Apache Spark) > TRANSFORM when schema less should keep same

[jira] [Reopened] (SPARK-32388) TRANSFORM when schema less should keep same with hive

2020-10-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-32388: -- > TRANSFORM when schema less should keep same with hive >

[jira] [Updated] (SPARK-32388) TRANSFORM when schema less should keep same with hive

2020-10-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32388: - Fix Version/s: (was: 3.1.0) > TRANSFORM when schema less should keep same with hive >

[jira] [Assigned] (SPARK-32388) TRANSFORM when schema less should keep same with hive

2020-10-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32388: Assignee: angerszhu > TRANSFORM when schema less should keep same with hive >

[jira] [Resolved] (SPARK-32388) TRANSFORM when schema less should keep same with hive

2020-10-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32388. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29421

[jira] [Commented] (SPARK-33237) Use default Hadoop profile by removing explicit `-Phadoop-2.7` from K8s IT Jenkins job

2020-10-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17220424#comment-17220424 ] Dongjoon Hyun commented on SPARK-33237: --- Hi, [~shaneknapp]. Could you remove `-Phadoop-2.7` in the

[jira] [Created] (SPARK-33237) Use default Hadoop profile by removing explicit `-Phadoop-2.7` from K8s IT Jenkins job

2020-10-25 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-33237: - Summary: Use default Hadoop profile by removing explicit `-Phadoop-2.7` from K8s IT Jenkins job Key: SPARK-33237 URL: https://issues.apache.org/jira/browse/SPARK-33237

[jira] [Created] (SPARK-33236) Enable Push-based shuffle service to store state in NM level DB for work preserving restart

2020-10-25 Thread Chandni Singh (Jira)
Chandni Singh created SPARK-33236: - Summary: Enable Push-based shuffle service to store state in NM level DB for work preserving restart Key: SPARK-33236 URL: https://issues.apache.org/jira/browse/SPARK-33236

[jira] [Updated] (SPARK-33235) Push-based Shuffle Phase 2 Tasks

2020-10-25 Thread Chandni Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated SPARK-33235: -- Description: This is the parent jira for the phase 2 or follow-up tasks for supporting

[jira] [Created] (SPARK-33235) Push-based Shuffle Phase 2 Tasks

2020-10-25 Thread Chandni Singh (Jira)
Chandni Singh created SPARK-33235: - Summary: Push-based Shuffle Phase 2 Tasks Key: SPARK-33235 URL: https://issues.apache.org/jira/browse/SPARK-33235 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-33234) Generates SHA-512 using shasum

2020-10-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-33234. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30123

[jira] [Assigned] (SPARK-33234) Generates SHA-512 using shasum

2020-10-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33234: Assignee: (was: Apache Spark) > Generates SHA-512 using shasum >

[jira] [Commented] (SPARK-33234) Generates SHA-512 using shasum

2020-10-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17220415#comment-17220415 ] Apache Spark commented on SPARK-33234: -- User 'emilianbold' has created a pull request for this

[jira] [Commented] (SPARK-33234) Generates SHA-512 using shasum

2020-10-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17220414#comment-17220414 ] Apache Spark commented on SPARK-33234: -- User 'emilianbold' has created a pull request for this

[jira] [Assigned] (SPARK-33234) Generates SHA-512 using shasum

2020-10-25 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33234: Assignee: Apache Spark > Generates SHA-512 using shasum > --

[jira] [Created] (SPARK-33234) Generates SHA-512 using shasum

2020-10-25 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-33234: - Summary: Generates SHA-512 using shasum Key: SPARK-33234 URL: https://issues.apache.org/jira/browse/SPARK-33234 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-33228) Don't uncache data when replacing an existing view having the same plan

2020-10-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-33228. --- Fix Version/s: 2.4.8 3.0.2 3.1.0 Resolution:

[jira] [Assigned] (SPARK-33228) Don't uncache data when replacing an existing view having the same plan

2020-10-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-33228: - Assignee: Takeshi Yamamuro > Don't uncache data when replacing an existing view having

[jira] [Resolved] (SPARK-31342) Fail by default if Parquet DATE or TIMESTAMP data is before October 15, 1582

2020-10-25 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins resolved SPARK-31342. --- Resolution: Duplicate > Fail by default if Parquet DATE or TIMESTAMP data is before October

[jira] [Commented] (SPARK-31342) Fail by default if Parquet DATE or TIMESTAMP data is before October 15, 1582

2020-10-25 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17220274#comment-17220274 ] Maxim Gekk commented on SPARK-31342: [~cloud_fan] I think we can close this because it has been

[jira] [Updated] (SPARK-33195) stages/stage UI page fails to load when spark reverse proxy is enabled

2020-10-25 Thread Liran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liran updated SPARK-33195: -- Description: I think we have the same issue reported in SPARK-32467, reproduced with reverse proxy