[jira] [Updated] (SPARK-34843) JDBCRelation columnPartition function improperly determines stride size

2021-03-24 Thread Jason Yarbrough (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Yarbrough updated SPARK-34843: Description: Currently, in JDBCRelation (line 123), the stride size is calculated as

[jira] [Comment Edited] (SPARK-34843) JDBCRelation columnPartition function improperly determines stride size

2021-03-24 Thread Jason Yarbrough (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308343#comment-17308343 ] Jason Yarbrough edited comment on SPARK-34843 at 3/25/21, 4:21 AM: ---

[jira] [Commented] (SPARK-34843) JDBCRelation columnPartition function improperly determines stride size

2021-03-24 Thread Jason Yarbrough (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308343#comment-17308343 ] Jason Yarbrough commented on SPARK-34843: - I've modified the formula slightly (reflected in the

[jira] [Updated] (SPARK-34843) JDBCRelation columnPartition function improperly determines stride size

2021-03-24 Thread Jason Yarbrough (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Yarbrough updated SPARK-34843: Description: Currently, in JDBCRelation (line 123), the stride size is calculated as

[jira] [Updated] (SPARK-34822) Update plan stability golden files even if only explain differs

2021-03-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-34822: - Fix Version/s: (was: 3.2.0) > Update plan stability golden files even if only explain

[jira] [Commented] (SPARK-34822) Update plan stability golden files even if only explain differs

2021-03-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308324#comment-17308324 ] Hyukjin Kwon commented on SPARK-34822: -- Reverted at 

[jira] [Updated] (SPARK-34855) SparkContext - avoid using local lazy val

2021-03-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-34855: - Issue Type: Bug (was: Improvement) > SparkContext - avoid using local lazy val >

[jira] [Created] (SPARK-34860) Multinomial Logistic Regression with intercept support centering

2021-03-24 Thread zhengruifeng (Jira)
zhengruifeng created SPARK-34860: Summary: Multinomial Logistic Regression with intercept support centering Key: SPARK-34860 URL: https://issues.apache.org/jira/browse/SPARK-34860 Project: Spark

[jira] [Assigned] (SPARK-34857) AtLeastNNonNulls does not show up correctly in explain

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34857: Assignee: (was: Apache Spark) > AtLeastNNonNulls does not show up correctly in

[jira] [Assigned] (SPARK-34857) AtLeastNNonNulls does not show up correctly in explain

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34857: Assignee: Apache Spark > AtLeastNNonNulls does not show up correctly in explain >

[jira] [Commented] (SPARK-34857) AtLeastNNonNulls does not show up correctly in explain

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308313#comment-17308313 ] Apache Spark commented on SPARK-34857: -- User 'timarmstrong' has created a pull request for this

[jira] [Assigned] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API call for a specified stage

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-34488: Assignee: angerszhu (was: Ron Hu) > Support task Metrics Distributions and executor

[jira] [Assigned] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API call for a specified stage

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-34488: Assignee: angerszhu (was: angerszhu) > Support task Metrics Distributions and executor

[jira] [Commented] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API call for a specified stage

2021-03-24 Thread Ron Hu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308305#comment-17308305 ] Ron Hu commented on SPARK-34488: Hi [~srowen], Thanks for merging this PR.  [~angerszhuuu] and I worked

[jira] [Commented] (SPARK-34852) Close Hive session state should use withHiveState

2021-03-24 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308295#comment-17308295 ] Kent Yao commented on SPARK-34852: -- Issue fixed by https://github.com/apache/spark/pull/31949 > Close

[jira] [Resolved] (SPARK-34852) Close Hive session state should use withHiveState

2021-03-24 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-34852. -- Fix Version/s: 3.2.0 Assignee: ulysses you Resolution: Fixed > Close Hive session

[jira] [Commented] (SPARK-26345) Parquet support Column indexes

2021-03-24 Thread Li Xian (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308292#comment-17308292 ] Li Xian commented on SPARK-26345: - [~dongjoon] sure, I have created a new issue

[jira] [Created] (SPARK-34859) Vectorized parquet reader needs synchronization among pages for column index

2021-03-24 Thread Li Xian (Jira)
Li Xian created SPARK-34859: --- Summary: Vectorized parquet reader needs synchronization among pages for column index Key: SPARK-34859 URL: https://issues.apache.org/jira/browse/SPARK-34859 Project: Spark

[jira] [Commented] (SPARK-34858) Binary Logistic Regression with intercept support centering

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308272#comment-17308272 ] Apache Spark commented on SPARK-34858: -- User 'zhengruifeng' has created a pull request for this

[jira] [Assigned] (SPARK-34858) Binary Logistic Regression with intercept support centering

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34858: Assignee: (was: Apache Spark) > Binary Logistic Regression with intercept support

[jira] [Assigned] (SPARK-34858) Binary Logistic Regression with intercept support centering

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34858: Assignee: Apache Spark > Binary Logistic Regression with intercept support centering >

[jira] [Created] (SPARK-34858) Binary Logistic Regression with intercept support centering

2021-03-24 Thread zhengruifeng (Jira)
zhengruifeng created SPARK-34858: Summary: Binary Logistic Regression with intercept support centering Key: SPARK-34858 URL: https://issues.apache.org/jira/browse/SPARK-34858 Project: Spark

[jira] [Commented] (SPARK-34780) Cached Table (parquet) with old Configs Used

2021-03-24 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308262#comment-17308262 ] Chao Sun commented on SPARK-34780: -- Sorry for the late reply [~mikechen]! There's something I still not

[jira] [Updated] (SPARK-34857) AtLeastNNonNulls does not show up correctly in explain

2021-03-24 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-34857: - Affects Version/s: (was: 3.1.1) 3.1.2

[jira] [Commented] (SPARK-34857) AtLeastNNonNulls does not show up correctly in explain

2021-03-24 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308257#comment-17308257 ] Takeshi Yamamuro commented on SPARK-34857: -- Please feel free to make a PR for this issue. >

[jira] [Updated] (SPARK-34829) transform_values return identical values when it's used with udf that returns reference type

2021-03-24 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-34829: - Labels: corre (was: ) > transform_values return identical values when it's used with

[jira] [Updated] (SPARK-34829) transform_values return identical values when it's used with udf that returns reference type

2021-03-24 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-34829: - Affects Version/s: 3.2.0 > transform_values return identical values when it's used with

[jira] [Updated] (SPARK-34829) transform_values return identical values when it's used with udf that returns reference type

2021-03-24 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-34829: - Labels: (was: corre) > transform_values return identical values when it's used with

[jira] [Updated] (SPARK-34829) transform_values return identical values when it's used with udf that returns reference type

2021-03-24 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-34829: - Labels: correctness (was: ) > transform_values return identical values when it's used

[jira] [Comment Edited] (SPARK-34844) JDBCRelation columnPartition function includes the first stride in the lower partition

2021-03-24 Thread Jason Yarbrough (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308249#comment-17308249 ] Jason Yarbrough edited comment on SPARK-34844 at 3/25/21, 12:19 AM:

[jira] [Resolved] (SPARK-34844) JDBCRelation columnPartition function includes the first stride in the lower partition

2021-03-24 Thread Jason Yarbrough (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Yarbrough resolved SPARK-34844. - Resolution: Not A Problem > JDBCRelation columnPartition function includes the first

[jira] [Commented] (SPARK-34844) JDBCRelation columnPartition function includes the first stride in the lower partition

2021-03-24 Thread Jason Yarbrough (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308249#comment-17308249 ] Jason Yarbrough commented on SPARK-34844: - As I coded up the unit tests and checked the effect

[jira] [Resolved] (SPARK-34833) Apply right-padding correctly for correlated subqueries

2021-03-24 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-34833. -- Fix Version/s: 3.1.2 3.2.0 Assignee: Takeshi Yamamuro

[jira] [Updated] (SPARK-34843) JDBCRelation columnPartition function improperly determines stride size

2021-03-24 Thread Jason Yarbrough (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Yarbrough updated SPARK-34843: Description: Currently, in JDBCRelation (line 123), the stride size is calculated as

[jira] [Updated] (SPARK-34843) JDBCRelation columnPartition function improperly determines stride size

2021-03-24 Thread Jason Yarbrough (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Yarbrough updated SPARK-34843: Description: Currently, in JDBCRelation (line 123), the stride size is calculated as

[jira] [Updated] (SPARK-34843) JDBCRelation columnPartition function improperly determines stride size

2021-03-24 Thread Jason Yarbrough (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Yarbrough updated SPARK-34843: Description: Currently, in JDBCRelation (line 123), the stride size is calculated as

[jira] [Commented] (SPARK-26345) Parquet support Column indexes

2021-03-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308144#comment-17308144 ] Dongjoon Hyun commented on SPARK-26345: --- Thank you for reporting. Could you file a new JIRA,

[jira] [Commented] (SPARK-30497) migrate DESCRIBE TABLE to the new framework

2021-03-24 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308067#comment-17308067 ] Chao Sun commented on SPARK-30497: -- [~cloud_fan] this is resolved right? > migrate DESCRIBE TABLE to

[jira] [Updated] (SPARK-34857) AtLeastNNonNulls does not show up correctly in explain

2021-03-24 Thread Tim Armstrong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated SPARK-34857: -- Priority: Minor (was: Major) > AtLeastNNonNulls does not show up correctly in explain >

[jira] [Created] (SPARK-34857) AtLeastNNonNulls does not show up correctly in explain

2021-03-24 Thread Tim Armstrong (Jira)
Tim Armstrong created SPARK-34857: - Summary: AtLeastNNonNulls does not show up correctly in explain Key: SPARK-34857 URL: https://issues.apache.org/jira/browse/SPARK-34857 Project: Spark

[jira] [Commented] (SPARK-34857) AtLeastNNonNulls does not show up correctly in explain

2021-03-24 Thread Tim Armstrong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308055#comment-17308055 ] Tim Armstrong commented on SPARK-34857: --- [~LI,Xiao] i would like to take this one on. >

[jira] [Updated] (SPARK-34857) AtLeastNNonNulls does not show up correctly in explain

2021-03-24 Thread Tim Armstrong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated SPARK-34857: -- Description: I see this in an explain plan {noformat} (12) Filter Input [3]: [c1#2410L,

[jira] [Updated] (SPARK-34857) AtLeastNNonNulls does not show up correctly in explain

2021-03-24 Thread Tim Armstrong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated SPARK-34857: -- Environment: (was: I see this in an explain plan {noformat} (12) Filter Input [3]:

[jira] [Updated] (SPARK-34596) NewInstance.doGenCode should not throw malformed class name error

2021-03-24 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-34596: Fix Version/s: 2.4.8 > NewInstance.doGenCode should not throw malformed class name error >

[jira] [Assigned] (SPARK-34596) NewInstance.doGenCode should not throw malformed class name error

2021-03-24 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-34596: --- Assignee: Kris Mok > NewInstance.doGenCode should not throw malformed class name error >

[jira] [Assigned] (SPARK-34829) transform_values return identical values when it's used with udf that returns reference type

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34829: Assignee: Apache Spark > transform_values return identical values when it's used with

[jira] [Assigned] (SPARK-34829) transform_values return identical values when it's used with udf that returns reference type

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34829: Assignee: (was: Apache Spark) > transform_values return identical values when it's

[jira] [Commented] (SPARK-34829) transform_values return identical values when it's used with udf that returns reference type

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308034#comment-17308034 ] Apache Spark commented on SPARK-34829: -- User 'peter-toth' has created a pull request for this

[jira] [Resolved] (SPARK-34797) Refactor Logistic Aggregator - support virtual centering

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-34797. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31889

[jira] [Assigned] (SPARK-34797) Refactor Logistic Aggregator - support virtual centering

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-34797: Assignee: zhengruifeng (was: Apache Spark) > Refactor Logistic Aggregator - support

[jira] [Commented] (SPARK-26345) Parquet support Column indexes

2021-03-24 Thread Xinli Shang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308000#comment-17308000 ] Xinli Shang commented on SPARK-26345: - Yes, it needs some synchronization. I have the modified

[jira] [Assigned] (SPARK-34856) ANSI mode: Allow casting complex types as string type

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34856: Assignee: Gengliang Wang (was: Apache Spark) > ANSI mode: Allow casting complex types

[jira] [Assigned] (SPARK-34856) ANSI mode: Allow casting complex types as string type

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34856: Assignee: Apache Spark (was: Gengliang Wang) > ANSI mode: Allow casting complex types

[jira] [Commented] (SPARK-34856) ANSI mode: Allow casting complex types as string type

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307910#comment-17307910 ] Apache Spark commented on SPARK-34856: -- User 'gengliangwang' has created a pull request for this

[jira] [Created] (SPARK-34856) ANSI mode: Allow casting complex types as string type

2021-03-24 Thread Gengliang Wang (Jira)
Gengliang Wang created SPARK-34856: -- Summary: ANSI mode: Allow casting complex types as string type Key: SPARK-34856 URL: https://issues.apache.org/jira/browse/SPARK-34856 Project: Spark

[jira] [Updated] (SPARK-33482) V2 Datasources that extend FileScan preclude exchange reuse

2021-03-24 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-33482: Fix Version/s: 3.0.3 > V2 Datasources that extend FileScan preclude exchange reuse >

[jira] [Updated] (SPARK-34756) Fix FileScan equality check

2021-03-24 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-34756: Fix Version/s: 3.0.3 > Fix FileScan equality check > --- > >

[jira] [Resolved] (SPARK-34769) AnsiTypeCoercion: return narrowest convertible type among TypeCollection

2021-03-24 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-34769. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31859

[jira] [Commented] (SPARK-34510) .foreachPartition command hangs when ran inside Python package but works when ran from Python file outside the package on EMR

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307879#comment-17307879 ] Sean R. Owen commented on SPARK-34510: -- Firstly, does this happen on Apache Spark or just EMR? Are

[jira] [Commented] (SPARK-34530) logError for interrupting block migrations is too high

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307878#comment-17307878 ] Sean R. Owen commented on SPARK-34530: -- Is this just about turning log level down? that'd be fine

[jira] [Commented] (SPARK-34569) java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307876#comment-17307876 ] Sean R. Owen commented on SPARK-34569: -- This sounds like a Spark version mismatch? >

[jira] [Updated] (SPARK-34591) Pyspark undertakes pruning of decision trees and random forests outside the control of the user, leading to undesirable and unexpected outcomes that are challenging to d

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-34591: - Priority: Minor (was: Critical) (Not nearly Critical) Yes please open a PR to add the

[jira] [Resolved] (SPARK-34822) Update plan stability golden files even if only explain differs

2021-03-24 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-34822. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31927

[jira] [Assigned] (SPARK-34822) Update plan stability golden files even if only explain differs

2021-03-24 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-34822: --- Assignee: Tanel Kiis > Update plan stability golden files even if only explain differs >

[jira] [Resolved] (SPARK-34630) Add type hints of pyspark.__version__ and pyspark.sql.Column.contains

2021-03-24 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-34630. Fix Version/s: 3.1.2 3.2.0 Assignee: Danny Meijer

[jira] [Resolved] (SPARK-34689) Spark Thrift Server: Memory leak for SparkSession objects

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-34689. -- Resolution: Duplicate > Spark Thrift Server: Memory leak for SparkSession objects >

[jira] [Resolved] (SPARK-34746) Spark dependencies require scala 2.12.12

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-34746. -- Resolution: Duplicate Yep, though there are reasons we couldn't use 2.12.12 > Spark

[jira] [Updated] (SPARK-34757) Spark submit should ignore cache for SNAPSHOT dependencies

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-34757: - Priority: Minor (was: Major) > Spark submit should ignore cache for SNAPSHOT dependencies >

[jira] [Resolved] (SPARK-34759) run JavaSparkSQLExample failed with Exception.

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-34759. -- Resolution: Duplicate > run JavaSparkSQLExample failed with Exception. >

[jira] [Resolved] (SPARK-34801) java.lang.NoSuchMethodException: org.apache.hadoop.hive.ql.metadata.Hive.loadPartition

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-34801. -- Resolution: Not A Problem > java.lang.NoSuchMethodException: >

[jira] [Updated] (SPARK-34838) consider markdownlint

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-34838: - Issue Type: Improvement (was: Bug) > consider markdownlint > - > >

[jira] [Resolved] (SPARK-34853) Move partitioning and ordering to common limit trait

2021-03-24 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-34853. -- Fix Version/s: 3.2.0 Assignee: Cheng Su Resolution: Fixed Resolved by

[jira] [Assigned] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API call for a specified stage

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-34488: Assignee: Ron Hu > Support task Metrics Distributions and executor Metrics Distributions

[jira] [Resolved] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API call for a specified stage

2021-03-24 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-34488. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31611

[jira] [Commented] (SPARK-34855) SparkContext - avoid using local lazy val

2021-03-24 Thread Li Xian (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307717#comment-17307717 ] Li Xian commented on SPARK-34855: - [https://github.com/apache/spark/pull/31953] I have submitted a PR to

[jira] [Assigned] (SPARK-34855) SparkContext - avoid using local lazy val

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34855: Assignee: (was: Apache Spark) > SparkContext - avoid using local lazy val >

[jira] [Assigned] (SPARK-34855) SparkContext - avoid using local lazy val

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34855: Assignee: Apache Spark > SparkContext - avoid using local lazy val >

[jira] [Commented] (SPARK-34855) SparkContext - avoid using local lazy val

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307718#comment-17307718 ] Apache Spark commented on SPARK-34855: -- User 'lxian' has created a pull request for this issue:

[jira] [Created] (SPARK-34855) SparkContext - avoid using local lazy val

2021-03-24 Thread Li Xian (Jira)
Li Xian created SPARK-34855: --- Summary: SparkContext - avoid using local lazy val Key: SPARK-34855 URL: https://issues.apache.org/jira/browse/SPARK-34855 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-34855) SparkContext - avoid using local lazy val

2021-03-24 Thread Li Xian (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Xian updated SPARK-34855: Attachment: Screen Shot 2021-03-24 at 5.41.22 PM.png > SparkContext - avoid using local lazy val >

[jira] [Commented] (SPARK-34684) Hadoop config could not be successfully serilized from driver pods to executor pods

2021-03-24 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307710#comment-17307710 ] Attila Zsolt Piros commented on SPARK-34684: I did not say there is no error but by

[jira] [Commented] (SPARK-34756) Fix FileScan equality check

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307702#comment-17307702 ] Apache Spark commented on SPARK-34756: -- User 'peter-toth' has created a pull request for this

[jira] [Commented] (SPARK-34756) Fix FileScan equality check

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307700#comment-17307700 ] Apache Spark commented on SPARK-34756: -- User 'peter-toth' has created a pull request for this

[jira] [Commented] (SPARK-34756) Fix FileScan equality check

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307699#comment-17307699 ] Apache Spark commented on SPARK-34756: -- User 'peter-toth' has created a pull request for this

[jira] [Commented] (SPARK-33482) V2 Datasources that extend FileScan preclude exchange reuse

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307698#comment-17307698 ] Apache Spark commented on SPARK-33482: -- User 'peter-toth' has created a pull request for this

[jira] [Updated] (SPARK-30641) Project Matrix: Linear Models revisit and refactor

2021-03-24 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-30641: - Description: We had been refactoring linear models for a long time, and there still are some

[jira] [Commented] (SPARK-33482) V2 Datasources that extend FileScan preclude exchange reuse

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307697#comment-17307697 ] Apache Spark commented on SPARK-33482: -- User 'peter-toth' has created a pull request for this

[jira] [Commented] (SPARK-30641) Project Matrix: Linear Models revisit and refactor

2021-03-24 Thread Weichen Xu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307690#comment-17307690 ] Weichen Xu commented on SPARK-30641: Good work! > Project Matrix: Linear Models revisit and

[jira] [Commented] (SPARK-34684) Hadoop config could not be successfully serilized from driver pods to executor pods

2021-03-24 Thread Yue Peng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307686#comment-17307686 ] Yue Peng commented on SPARK-34684: -- Why we need to do this? Accessing files from executor should be by

[jira] [Assigned] (SPARK-34850) Multiply day-time interval by numeric

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34850: Assignee: Apache Spark (was: Max Gekk) > Multiply day-time interval by numeric >

[jira] [Assigned] (SPARK-34850) Multiply day-time interval by numeric

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34850: Assignee: Max Gekk (was: Apache Spark) > Multiply day-time interval by numeric >

[jira] [Commented] (SPARK-34850) Multiply day-time interval by numeric

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307655#comment-17307655 ] Apache Spark commented on SPARK-34850: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Updated] (SPARK-34345) Allow several properties files

2021-03-24 Thread Arseniy Tashoyan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arseniy Tashoyan updated SPARK-34345: - Component/s: Kubernetes > Allow several properties files >

[jira] [Commented] (SPARK-34345) Allow several properties files

2021-03-24 Thread Arseniy Tashoyan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307644#comment-17307644 ] Arseniy Tashoyan commented on SPARK-34345: -- Typesafe Config seems attractive, as it provides

[jira] [Resolved] (SPARK-34295) Allow option similar to mapreduce.job.hdfs-servers.token-renewal.exclude

2021-03-24 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-34295. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31761

[jira] [Updated] (SPARK-34851) Add scope tag in configuration

2021-03-24 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-34851: -- Description: According to discussion at

[jira] [Updated] (SPARK-34345) Allow several properties files

2021-03-24 Thread Arseniy Tashoyan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arseniy Tashoyan updated SPARK-34345: - Affects Version/s: 3.1.1 > Allow several properties files >

[jira] [Updated] (SPARK-34345) Allow several properties files

2021-03-24 Thread Arseniy Tashoyan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arseniy Tashoyan updated SPARK-34345: - Priority: Major (was: Minor) > Allow several properties files >

[jira] [Updated] (SPARK-31531) sun.misc.Cleaner sun.nio.ch.DirectBuffer.cleaner() method not found during spark-submit

2021-03-24 Thread Arseniy Tashoyan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arseniy Tashoyan updated SPARK-31531: - Attachment: image.png > sun.misc.Cleaner sun.nio.ch.DirectBuffer.cleaner() method not

[jira] [Commented] (SPARK-34854) Report metrics for streaming source through progress reporter with Kafka source use-case

2021-03-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307631#comment-17307631 ] Apache Spark commented on SPARK-34854: -- User 'yijiacui-db' has created a pull request for this

  1   2   >