[jira] [Commented] (SPARK-35264) Support AQE side broadcastJoin threshold

2021-05-05 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17340026#comment-17340026 ] Dongjoon Hyun commented on SPARK-35264: --- Thank YOU, [~ulysses]! :) > Support AQE

[jira] [Updated] (SPARK-31325) Control a plan explain mode in the events of SQL listeners via SQLConf

2021-05-05 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-31325: - Fix Version/s: 3.1.0 > Control a plan explain mode in the events of SQL listeners via SQ

[jira] [Resolved] (SPARK-35293) Use the newer dsdgen for TPCDSQueryTestSuite

2021-05-05 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-35293. -- Fix Version/s: 3.2.0 Resolution: Fixed Resolved by https://github.com/apache/sp

[jira] [Assigned] (SPARK-35282) Support AQE side shuffled hash join formula

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35282: Assignee: (was: Apache Spark) > Support AQE side shuffled hash join formula > ---

[jira] [Assigned] (SPARK-35282) Support AQE side shuffled hash join formula

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35282: Assignee: Apache Spark > Support AQE side shuffled hash join formula > --

[jira] [Commented] (SPARK-35282) Support AQE side shuffled hash join formula

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17340004#comment-17340004 ] Apache Spark commented on SPARK-35282: -- User 'ulysses-you' has created a pull reque

[jira] [Updated] (SPARK-35282) Support AQE side shuffled hash join formula

2021-05-05 Thread ulysses you (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ulysses you updated SPARK-35282: Description: Use AQE runtime statistics to decide if we can use shuffled hash join instead of sor

[jira] [Resolved] (SPARK-35325) Add nested column ORC encryption test case

2021-05-05 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-35325. --- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32449 [https://

[jira] [Assigned] (SPARK-35325) Add nested column ORC encryption test case

2021-05-05 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-35325: - Assignee: Dongjoon Hyun > Add nested column ORC encryption test case >

[jira] [Updated] (SPARK-35144) Migrate to transformWithPruning or resolveWithPruning for object rules

2021-05-05 Thread Yingyi Bu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingyi Bu updated SPARK-35144: -- Description: Rules in org/apache/spark/sql/catalyst/optimizer/objects.Scala   Commit  [https://githu

[jira] [Updated] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing

2021-05-05 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-35321: - Issue Type: Bug (was: Improvement) > Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functio

[jira] [Assigned] (SPARK-35325) Add nested column ORC encryption test case

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35325: Assignee: (was: Apache Spark) > Add nested column ORC encryption test case >

[jira] [Assigned] (SPARK-35325) Add nested column ORC encryption test case

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35325: Assignee: Apache Spark > Add nested column ORC encryption test case > ---

[jira] [Commented] (SPARK-35325) Add nested column ORC encryption test case

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339974#comment-17339974 ] Apache Spark commented on SPARK-35325: -- User 'dongjoon-hyun' has created a pull req

[jira] [Created] (SPARK-35325) Add nested column ORC encryption test case

2021-05-05 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-35325: - Summary: Add nested column ORC encryption test case Key: SPARK-35325 URL: https://issues.apache.org/jira/browse/SPARK-35325 Project: Spark Issue Type: Impr

[jira] [Resolved] (SPARK-35319) Upgrade K8s client to 5.3.1

2021-05-05 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-35319. --- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32443 [https://

[jira] [Assigned] (SPARK-35319) Upgrade K8s client to 5.3.1

2021-05-05 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-35319: - Assignee: Dongjoon Hyun > Upgrade K8s client to 5.3.1 > --- > >

[jira] [Updated] (SPARK-34980) Support coalesce partition through union

2021-05-05 Thread ulysses you (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ulysses you updated SPARK-34980: Parent: SPARK-33828 Issue Type: Sub-task (was: Improvement) > Support coalesce partition

[jira] [Updated] (SPARK-35214) OptimizeSkewedJoin support ShuffledHashJoinExec

2021-05-05 Thread ulysses you (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ulysses you updated SPARK-35214: Parent: SPARK-33828 Issue Type: Sub-task (was: Improvement) > OptimizeSkewedJoin support

[jira] [Updated] (SPARK-35282) Support AQE side shuffled hash join formula

2021-05-05 Thread ulysses you (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ulysses you updated SPARK-35282: Parent: SPARK-33828 Issue Type: Sub-task (was: New Feature) > Support AQE side shuffled h

[jira] [Updated] (SPARK-35239) Coalesce shuffle partition should handle empty input RDD

2021-05-05 Thread ulysses you (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ulysses you updated SPARK-35239: Parent: SPARK-33828 Issue Type: Sub-task (was: Improvement) > Coalesce shuffle partition

[jira] [Commented] (SPARK-35290) unionByName with null filling fails for some nested structs

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339962#comment-17339962 ] Apache Spark commented on SPARK-35290: -- User 'Kimahriman' has created a pull reques

[jira] [Assigned] (SPARK-35290) unionByName with null filling fails for some nested structs

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35290: Assignee: Apache Spark > unionByName with null filling fails for some nested structs > --

[jira] [Assigned] (SPARK-35290) unionByName with null filling fails for some nested structs

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35290: Assignee: (was: Apache Spark) > unionByName with null filling fails for some nested s

[jira] [Commented] (SPARK-35290) unionByName with null filling fails for some nested structs

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339961#comment-17339961 ] Apache Spark commented on SPARK-35290: -- User 'Kimahriman' has created a pull reques

[jira] [Updated] (SPARK-34899) Use origin plan if can not coalesce shuffle partition

2021-05-05 Thread ulysses you (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ulysses you updated SPARK-34899: Parent: SPARK-33828 Issue Type: Sub-task (was: Improvement) > Use origin plan if can not

[jira] [Commented] (SPARK-34701) Remove analyzing temp view again in CreateViewCommand

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339959#comment-17339959 ] Apache Spark commented on SPARK-34701: -- User 'imback82' has created a pull request

[jira] [Commented] (SPARK-34701) Remove analyzing temp view again in CreateViewCommand

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339960#comment-17339960 ] Apache Spark commented on SPARK-34701: -- User 'imback82' has created a pull request

[jira] [Commented] (SPARK-35264) Support AQE side broadcastJoin threshold

2021-05-05 Thread ulysses you (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339958#comment-17339958 ] ulysses you commented on SPARK-35264: - [~dongjoon] thank you for moving this to AQE

[jira] [Resolved] (SPARK-35323) Remove unused libraries from LICENSE-binary

2021-05-05 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-35323. --- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32445 [https://

[jira] [Assigned] (SPARK-35323) Remove unused libraries from LICENSE-binary

2021-05-05 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-35323: - Assignee: Dongjoon Hyun > Remove unused libraries from LICENSE-binary > ---

[jira] [Resolved] (SPARK-35155) Add rule id to all Analyzer rules in fixed point batches

2021-05-05 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-35155. Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32425 [https:

[jira] [Assigned] (SPARK-35155) Add rule id to all Analyzer rules in fixed point batches

2021-05-05 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-35155: -- Assignee: Yingyi Bu > Add rule id to all Analyzer rules in fixed point batches >

[jira] [Commented] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339948#comment-17339948 ] Apache Spark commented on SPARK-35321: -- User 'sunchao' has created a pull request f

[jira] [Assigned] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35321: Assignee: (was: Apache Spark) > Spark 3.x can't talk to HMS 1.2.x and lower due to ge

[jira] [Assigned] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35321: Assignee: Apache Spark > Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_funct

[jira] [Commented] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339947#comment-17339947 ] Apache Spark commented on SPARK-35321: -- User 'sunchao' has created a pull request f

[jira] [Commented] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing

2021-05-05 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339946#comment-17339946 ] Chao Sun commented on SPARK-35321: -- [~yumwang] I'm thinking of using  [Hive#getWithFast

[jira] [Commented] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing

2021-05-05 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339944#comment-17339944 ] Yuming Wang commented on SPARK-35321: - Could we add a parameter to disable registerA

[jira] [Commented] (SPARK-35324) Time parser policy not respected in RDD

2021-05-05 Thread Willi Raschkowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339939#comment-17339939 ] Willi Raschkowski commented on SPARK-35324: --- This also reproduces if a launch

[jira] [Commented] (SPARK-35324) Time parser policy not respected in RDD

2021-05-05 Thread Willi Raschkowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339938#comment-17339938 ] Willi Raschkowski commented on SPARK-35324: --- I think the difference might have

[jira] [Updated] (SPARK-35324) Time parser policy not respected in RDD

2021-05-05 Thread Willi Raschkowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Willi Raschkowski updated SPARK-35324: -- Attachment: Screen Shot 2021-05-06 at 00.35.10.png Screen Shot 2021-05-

[jira] [Updated] (SPARK-35324) Time parser policy not respected in RDD

2021-05-05 Thread Willi Raschkowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Willi Raschkowski updated SPARK-35324: -- Description: When reading a CSV file, {{spark.sql.timeParserPolicy}} is respected in a

[jira] [Comment Edited] (SPARK-35324) Time parser policy not respected in RDD

2021-05-05 Thread Willi Raschkowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339933#comment-17339933 ] Willi Raschkowski edited comment on SPARK-35324 at 5/5/21, 11:26 PM: -

[jira] [Commented] (SPARK-35324) Time parser policy not respected in RDD

2021-05-05 Thread Willi Raschkowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339933#comment-17339933 ] Willi Raschkowski commented on SPARK-35324: --- For what it's worth, I only manag

[jira] [Commented] (SPARK-35324) Time parser policy not respected in RDD

2021-05-05 Thread Willi Raschkowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339931#comment-17339931 ] Willi Raschkowski commented on SPARK-35324: --- Same reading JSON: {code:title=t

[jira] [Updated] (SPARK-35324) Time parser policy not respected in RDD

2021-05-05 Thread Willi Raschkowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Willi Raschkowski updated SPARK-35324: -- Component/s: SQL > Time parser policy not respected in RDD > -

[jira] [Updated] (SPARK-35324) Time parser policy not respected in RDD

2021-05-05 Thread Willi Raschkowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Willi Raschkowski updated SPARK-35324: -- Issue Type: Bug (was: Task) > Time parser policy not respected in RDD > -

[jira] [Created] (SPARK-35324) Time parser policy not respected in RDD

2021-05-05 Thread Willi Raschkowski (Jira)
Willi Raschkowski created SPARK-35324: - Summary: Time parser policy not respected in RDD Key: SPARK-35324 URL: https://issues.apache.org/jira/browse/SPARK-35324 Project: Spark Issue Type:

[jira] [Issue Comment Deleted] (SPARK-35323) Remove unused libraries from LICENSE-binary

2021-05-05 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-35323: -- Comment: was deleted (was: User 'dongjoon-hyun' has created a pull request for this issue: htt

[jira] [Commented] (SPARK-35323) Remove unused libraries from LICENSE-binary

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339916#comment-17339916 ] Apache Spark commented on SPARK-35323: -- User 'dongjoon-hyun' has created a pull req

[jira] [Commented] (SPARK-35323) Remove unused libraries from LICENSE-binary

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339915#comment-17339915 ] Apache Spark commented on SPARK-35323: -- User 'dongjoon-hyun' has created a pull req

[jira] [Assigned] (SPARK-35323) Remove unused libraries from LICENSE-binary

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35323: Assignee: Apache Spark > Remove unused libraries from LICENSE-binary > --

[jira] [Assigned] (SPARK-35323) Remove unused libraries from LICENSE-binary

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35323: Assignee: (was: Apache Spark) > Remove unused libraries from LICENSE-binary > ---

[jira] [Updated] (SPARK-35323) Remove unused libraries from LICENSE-binary

2021-05-05 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-35323: -- Summary: Remove unused libraries from LICENSE-binary (was: Remove unused libraries from LICEN

[jira] [Created] (SPARK-35323) Remove unused libraries from LICENSE

2021-05-05 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-35323: - Summary: Remove unused libraries from LICENSE Key: SPARK-35323 URL: https://issues.apache.org/jira/browse/SPARK-35323 Project: Spark Issue Type: Improvemen

[jira] [Commented] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing

2021-05-05 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339911#comment-17339911 ] Erik Krogen commented on SPARK-35321: - Gotcha. Yeah, agreed that if it's unnecessary

[jira] [Created] (SPARK-35322) Unable to Connect to SSL enabled Oracle RDS Instance using pyspark with spark-submit cluster mode

2021-05-05 Thread Bhragu (Jira)
Bhragu created SPARK-35322: -- Summary: Unable to Connect to SSL enabled Oracle RDS Instance using pyspark with spark-submit cluster mode Key: SPARK-35322 URL: https://issues.apache.org/jira/browse/SPARK-35322

[jira] [Commented] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing

2021-05-05 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339906#comment-17339906 ] Chao Sun commented on SPARK-35321: -- [~xkrogen] yes that can help to solve the issue, bu

[jira] [Commented] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing

2021-05-05 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339904#comment-17339904 ] Erik Krogen commented on SPARK-35321: - Isn't this what the {{IsolatedClientLoader}}

[jira] [Commented] (SPARK-34511) Current Security vulnerabilities in spark libraries

2021-05-05 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339902#comment-17339902 ] Dongjoon Hyun commented on SPARK-34511: --- Also, cc [~sunchao] , too. > Current Sec

[jira] [Created] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing

2021-05-05 Thread Chao Sun (Jira)
Chao Sun created SPARK-35321: Summary: Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing Key: SPARK-35321 URL: https://issues.apache.org/jira/browse/SPARK-35321 Proje

[jira] [Updated] (SPARK-35321) Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing

2021-05-05 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-35321: - Description: https://issues.apache.org/jira/browse/HIVE-10319 introduced a new API {{get_all_functions}

[jira] [Commented] (SPARK-34511) Current Security vulnerabilities in spark libraries

2021-05-05 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339899#comment-17339899 ] Dongjoon Hyun commented on SPARK-34511: --- Hi, [~harg].  Apache Spark master branch

[jira] [Commented] (SPARK-34791) SparkR throws node stack overflow

2021-05-05 Thread obfuscated_dvlper (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339890#comment-17339890 ] obfuscated_dvlper commented on SPARK-34791: --- tried R-4.0.3 as well, didn't hel

[jira] [Commented] (SPARK-34941) Enable mypy for pandas-on-Spark

2021-05-05 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339868#comment-17339868 ] Takuya Ueshin commented on SPARK-34941: --- I'm going to work on this. > Enable mypy

[jira] [Updated] (SPARK-35320) from_json cannot parse maps with timestamp as key

2021-05-05 Thread Vincenzo Cerminara (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincenzo Cerminara updated SPARK-35320: --- Description: I have a json that contains a {{map}} like the following {code:json} {

[jira] [Created] (SPARK-35320) from_json cannot parse maps with timestamp as key

2021-05-05 Thread Vincenzo Cerminara (Jira)
Vincenzo Cerminara created SPARK-35320: -- Summary: from_json cannot parse maps with timestamp as key Key: SPARK-35320 URL: https://issues.apache.org/jira/browse/SPARK-35320 Project: Spark

[jira] [Issue Comment Deleted] (SPARK-35319) Upgrade K8s client to 5.3.1

2021-05-05 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-35319: -- Comment: was deleted (was: User 'dongjoon-hyun' has created a pull request for this issue: htt

[jira] [Commented] (SPARK-35111) Cast string to year-month interval

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339745#comment-17339745 ] Apache Spark commented on SPARK-35111: -- User 'AngersZh' has created a pull requ

[jira] [Commented] (SPARK-35319) Upgrade K8s client to 5.3.1

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339733#comment-17339733 ] Apache Spark commented on SPARK-35319: -- User 'dongjoon-hyun' has created a pull req

[jira] [Assigned] (SPARK-35319) Upgrade K8s client to 5.3.1

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35319: Assignee: (was: Apache Spark) > Upgrade K8s client to 5.3.1 > ---

[jira] [Assigned] (SPARK-35319) Upgrade K8s client to 5.3.1

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35319: Assignee: Apache Spark > Upgrade K8s client to 5.3.1 > --- > >

[jira] [Commented] (SPARK-35319) Upgrade K8s client to 5.3.1

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339732#comment-17339732 ] Apache Spark commented on SPARK-35319: -- User 'dongjoon-hyun' has created a pull req

[jira] [Created] (SPARK-35319) Upgrade K8s client to 5.3.1

2021-05-05 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-35319: - Summary: Upgrade K8s client to 5.3.1 Key: SPARK-35319 URL: https://issues.apache.org/jira/browse/SPARK-35319 Project: Spark Issue Type: Bug Compo

[jira] [Commented] (SPARK-35283) Support query some DDL with CTES

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339628#comment-17339628 ] Apache Spark commented on SPARK-35283: -- User 'beliefer' has created a pull request

[jira] [Assigned] (SPARK-35283) Support query some DDL with CTES

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35283: Assignee: (was: Apache Spark) > Support query some DDL with CTES > --

[jira] [Assigned] (SPARK-35283) Support query some DDL with CTES

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35283: Assignee: Apache Spark > Support query some DDL with CTES > -

[jira] [Resolved] (SPARK-35315) Keep benchmark result consistent between spark-submit and SBT

2021-05-05 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-35315. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32440 [https://gith

[jira] [Assigned] (SPARK-35315) Keep benchmark result consistent between spark-submit and SBT

2021-05-05 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-35315: --- Assignee: Chao Sun > Keep benchmark result consistent between spark-submit and SBT > --

[jira] [Commented] (SPARK-35318) View internal properties should be hidden for describe table command

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339548#comment-17339548 ] Apache Spark commented on SPARK-35318: -- User 'linhongliu-db' has created a pull req

[jira] [Assigned] (SPARK-35318) View internal properties should be hidden for describe table command

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35318: Assignee: (was: Apache Spark) > View internal properties should be hidden for describ

[jira] [Assigned] (SPARK-35318) View internal properties should be hidden for describe table command

2021-05-05 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35318: Assignee: Apache Spark > View internal properties should be hidden for describe table com

[jira] [Created] (SPARK-35318) View internal properties should be hidden for describe table command

2021-05-05 Thread Linhong Liu (Jira)
Linhong Liu created SPARK-35318: --- Summary: View internal properties should be hidden for describe table command Key: SPARK-35318 URL: https://issues.apache.org/jira/browse/SPARK-35318 Project: Spark

[jira] [Resolved] (SPARK-34854) Report metrics for streaming source through progress reporter with Kafka source use-case

2021-05-05 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-34854. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31944 [https://gi

[jira] [Assigned] (SPARK-34854) Report metrics for streaming source through progress reporter with Kafka source use-case

2021-05-05 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-34854: Assignee: Yijia Cui > Report metrics for streaming source through progress reporter with

[jira] [Updated] (SPARK-35313) Hive MetaException attempting to get partition metadata by filter from Hive

2021-05-05 Thread Kaushik Muniandi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaushik Muniandi updated SPARK-35313: - Shepherd: Hyukjin Kwon (was: Andrew) > Hive MetaException attempting to get partition m

[jira] [Updated] (SPARK-35313) Hive MetaException attempting to get partition metadata by filter from Hive

2021-05-05 Thread Kaushik Muniandi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaushik Muniandi updated SPARK-35313: - Environment: Databricks runtime version 7.5 (includes Apache Spark 3.0.1, Scala 2.12) (

[jira] [Updated] (SPARK-35313) Hive MetaException attempting to get partition metadata by filter from Hive

2021-05-05 Thread Kaushik Muniandi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaushik Muniandi updated SPARK-35313: - Component/s: SQL > Hive MetaException attempting to get partition metadata by filter fro