[jira] [Created] (SPARK-32988) Spark 2.3 and 2.4 backward compatibility problems

dinesh created SPARK-32988: -- Summary: Spark 2.3 and 2.4 backward compatibility problems Key: SPARK-32988 URL: https://issues.apache.org/jira/browse/SPARK-32988 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-32989) Performance regression when selecting from str_to_map

Ondrej Kokes created SPARK-32989: Summary: Performance regression when selecting from str_to_map Key: SPARK-32989 URL: https://issues.apache.org/jira/browse/SPARK-32989 Project: Spark Issue

[jira] [Updated] (SPARK-32993) pyspark on yarn executor hang after some time

[ https://issues.apache.org/jira/browse/SPARK-32993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhao yufei updated SPARK-32993: --- Description: same pyspark program, previously it works ok, but these few days it always keep hang

[jira] [Updated] (SPARK-32988) ExternalCatalog vs ExternalCatalogWithListener: backward compatibility problem

[ https://issues.apache.org/jira/browse/SPARK-32988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dinesh updated SPARK-32988: --- Summary: ExternalCatalog vs ExternalCatalogWithListener: backward compatibility problem (was:

[jira] [Commented] (SPARK-32990) Migrate REFRESH TABLE to new resolution framework

[ https://issues.apache.org/jira/browse/SPARK-32990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201773#comment-17201773 ] Apache Spark commented on SPARK-32990: -- User 'imback82' has created a pull request for this issue:

[jira] [Commented] (SPARK-32990) Migrate REFRESH TABLE to new resolution framework

[ https://issues.apache.org/jira/browse/SPARK-32990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201772#comment-17201772 ] Apache Spark commented on SPARK-32990: -- User 'imback82' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32990) Migrate REFRESH TABLE to new resolution framework

[ https://issues.apache.org/jira/browse/SPARK-32990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32990: Assignee: Apache Spark > Migrate REFRESH TABLE to new resolution framework >

[jira] [Assigned] (SPARK-32990) Migrate REFRESH TABLE to new resolution framework

[ https://issues.apache.org/jira/browse/SPARK-32990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32990: Assignee: (was: Apache Spark) > Migrate REFRESH TABLE to new resolution framework >

[jira] [Created] (SPARK-32992) In OracleDialect, "RowID" SQL type should be converted into "String" Catalyst type

Peng Cheng created SPARK-32992: -- Summary: In OracleDialect, "RowID" SQL type should be converted into "String" Catalyst type Key: SPARK-32992 URL: https://issues.apache.org/jira/browse/SPARK-32992

[jira] [Updated] (SPARK-32993) pyspark on yarn executor hang after some time

[ https://issues.apache.org/jira/browse/SPARK-32993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhao yufei updated SPARK-32993: --- Attachment: driver.dump > pyspark on yarn executor hang after some time >

[jira] [Updated] (SPARK-32993) pyspark on yarn executor hang after some time

[ https://issues.apache.org/jira/browse/SPARK-32993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhao yufei updated SPARK-32993: --- Attachment: executor.dump > pyspark on yarn executor hang after some time >

[jira] [Updated] (SPARK-32993) pyspark on yarn executor hang after some time

[ https://issues.apache.org/jira/browse/SPARK-32993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhao yufei updated SPARK-32993: --- Description: same pyspark program, previously it works ok, but these few days it always keep hang

[jira] [Resolved] (SPARK-32993) pyspark on yarn executor hang after some time

[ https://issues.apache.org/jira/browse/SPARK-32993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhao yufei resolved SPARK-32993. Resolution: Won't Fix > pyspark on yarn executor hang after some time >

[jira] [Commented] (SPARK-32991) RESET can clear StaticSQLConfs

[ https://issues.apache.org/jira/browse/SPARK-32991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201869#comment-17201869 ] Kent Yao commented on SPARK-32991: -- I see the problem, I can fix it later, thanks for pinging me >

[jira] [Created] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

Lantao Jin created SPARK-32994: -- Summary: External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem Key: SPARK-32994 URL:

[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Attachment: Screen Shot 2020-09-24 at 5.19.58 PM.png > External accumulators (not start with

[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Attachment: Screen Shot 2020-09-24 at 5.19.26 PM.png > External accumulators (not start with

[jira] [Commented] (SPARK-32153) .m2 repository corruption happens

[ https://issues.apache.org/jira/browse/SPARK-32153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201832#comment-17201832 ] Kousuke Saruta commented on SPARK-32153: [~shaneknapp]Thank you so much! > .m2 repository

[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Description: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem

[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Description: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem

[jira] [Assigned] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32994: Assignee: Apache Spark > External accumulators (not start with

[jira] [Commented] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201893#comment-17201893 ] Apache Spark commented on SPARK-32994: -- User 'LantaoJin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32994: Assignee: (was: Apache Spark) > External accumulators (not start with

[jira] [Commented] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201892#comment-17201892 ] Apache Spark commented on SPARK-32994: -- User 'LantaoJin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32990) Migrate REFRESH TABLE to new resolution framework

[ https://issues.apache.org/jira/browse/SPARK-32990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-32990: --- Assignee: Terry Kim > Migrate REFRESH TABLE to new resolution framework >

[jira] [Resolved] (SPARK-32990) Migrate REFRESH TABLE to new resolution framework

[ https://issues.apache.org/jira/browse/SPARK-32990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-32990. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29866

[jira] [Resolved] (SPARK-32877) Fix Hive UDF not support decimal type in complex type

[ https://issues.apache.org/jira/browse/SPARK-32877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32877. --- Fix Version/s: 3.0.2 3.1.0 Resolution: Fixed Issue resolved by

[jira] [Assigned] (SPARK-32877) Fix Hive UDF not support decimal type in complex type

[ https://issues.apache.org/jira/browse/SPARK-32877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-32877: - Assignee: ulysses you > Fix Hive UDF not support decimal type in complex type >

[jira] [Commented] (SPARK-32973) FeatureHasher does not check categoricalCols in inputCols

[ https://issues.apache.org/jira/browse/SPARK-32973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201852#comment-17201852 ] zhengruifeng commented on SPARK-32973: -- yes, "real" is ignored here. Since it has been behaving

[jira] [Assigned] (SPARK-32973) FeatureHasher does not check categoricalCols in inputCols

[ https://issues.apache.org/jira/browse/SPARK-32973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32973: Assignee: (was: Apache Spark) > FeatureHasher does not check categoricalCols in

[jira] [Commented] (SPARK-32973) FeatureHasher does not check categoricalCols in inputCols

[ https://issues.apache.org/jira/browse/SPARK-32973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201882#comment-17201882 ] Apache Spark commented on SPARK-32973: -- User 'zhengruifeng' has created a pull request for this

[jira] [Assigned] (SPARK-32973) FeatureHasher does not check categoricalCols in inputCols

[ https://issues.apache.org/jira/browse/SPARK-32973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32973: Assignee: Apache Spark > FeatureHasher does not check categoricalCols in inputCols >

[jira] [Resolved] (SPARK-32153) .m2 repository corruption happens

[ https://issues.apache.org/jira/browse/SPARK-32153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved SPARK-32153. Resolution: Fixed > .m2 repository corruption happens > -

[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Attachment: Screen Shot 2020-09-25 at 11.36.48 AM.png > External accumulators (not start with

[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Attachment: Screen Shot 2020-09-25 at 11.32.51 AM.png > External accumulators (not start with

[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Attachment: Screen Shot 2020-09-25 at 11.35.01 AM.png > External accumulators (not start with

[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Description: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem

[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Description: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem

[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Description: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem

[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Description: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem

[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Description: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem

[jira] [Commented] (SPARK-32987) Pass all `mesos` module UTs in Scala 2.13

[ https://issues.apache.org/jira/browse/SPARK-32987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201454#comment-17201454 ] Apache Spark commented on SPARK-32987: -- User 'LuciferYang' has created a pull request for this

[jira] [Assigned] (SPARK-32987) Pass all `mesos` module UTs in Scala 2.13

[ https://issues.apache.org/jira/browse/SPARK-32987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32987: Assignee: Apache Spark > Pass all `mesos` module UTs in Scala 2.13 >

[jira] [Assigned] (SPARK-32987) Pass all `mesos` module UTs in Scala 2.13

[ https://issues.apache.org/jira/browse/SPARK-32987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32987: Assignee: (was: Apache Spark) > Pass all `mesos` module UTs in Scala 2.13 >

[jira] [Commented] (SPARK-32972) Pass all `mllib` module UTs in Scala 2.13

[ https://issues.apache.org/jira/browse/SPARK-32972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201452#comment-17201452 ] Apache Spark commented on SPARK-32972: -- User 'LuciferYang' has created a pull request for this

[jira] [Commented] (SPARK-32987) Pass all `mesos` module UTs in Scala 2.13

[ https://issues.apache.org/jira/browse/SPARK-32987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201453#comment-17201453 ] Apache Spark commented on SPARK-32987: -- User 'LuciferYang' has created a pull request for this

[jira] [Created] (SPARK-32985) Decouple bucket filter pruning and bucket table scan

Cheng Su created SPARK-32985: Summary: Decouple bucket filter pruning and bucket table scan Key: SPARK-32985 URL: https://issues.apache.org/jira/browse/SPARK-32985 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-32956) Duplicate Columns in a csv file

[ https://issues.apache.org/jira/browse/SPARK-32956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32956: Assignee: (was: Apache Spark) > Duplicate Columns in a csv file >

[jira] [Commented] (SPARK-32956) Duplicate Columns in a csv file

[ https://issues.apache.org/jira/browse/SPARK-32956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201422#comment-17201422 ] Apache Spark commented on SPARK-32956: -- User 'izchen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32956) Duplicate Columns in a csv file

[ https://issues.apache.org/jira/browse/SPARK-32956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32956: Assignee: Apache Spark > Duplicate Columns in a csv file >

[jira] [Commented] (SPARK-32877) Fix Hive UDF not support decimal type in complex type

[ https://issues.apache.org/jira/browse/SPARK-32877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201436#comment-17201436 ] Apache Spark commented on SPARK-32877: -- User 'ulysses-you' has created a pull request for this

[jira] [Commented] (SPARK-32877) Fix Hive UDF not support decimal type in complex type

[ https://issues.apache.org/jira/browse/SPARK-32877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201434#comment-17201434 ] Apache Spark commented on SPARK-32877: -- User 'ulysses-you' has created a pull request for this

[jira] [Commented] (SPARK-32987) Pass all `mesos` module UTs in Scala 2.13

[ https://issues.apache.org/jira/browse/SPARK-32987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201456#comment-17201456 ] Apache Spark commented on SPARK-32987: -- User 'LuciferYang' has created a pull request for this

[jira] [Commented] (SPARK-32987) Pass all `mesos` module UTs in Scala 2.13

[ https://issues.apache.org/jira/browse/SPARK-32987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201458#comment-17201458 ] Apache Spark commented on SPARK-32987: -- User 'LuciferYang' has created a pull request for this

[jira] [Commented] (SPARK-32956) Duplicate Columns in a csv file

[ https://issues.apache.org/jira/browse/SPARK-32956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201423#comment-17201423 ] Apache Spark commented on SPARK-32956: -- User 'izchen' has created a pull request for this issue:

[jira] [Created] (SPARK-32987) Pass all `mesos` module UTs in Scala 2.13

Yang Jie created SPARK-32987: Summary: Pass all `mesos` module UTs in Scala 2.13 Key: SPARK-32987 URL: https://issues.apache.org/jira/browse/SPARK-32987 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-32961) PySpark CSV read with UTF-16 encoding is not working correctly

[ https://issues.apache.org/jira/browse/SPARK-32961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201362#comment-17201362 ] Bui Bao Anh commented on SPARK-32961: - Hi [~hyukjin.kwon], got it,  For now we don't want to change

[jira] [Updated] (SPARK-32975) [K8S] - executor fails to be restarted after it goes to ERROR/Failure state

[ https://issues.apache.org/jira/browse/SPARK-32975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32975: - Priority: Major (was: Critical) > [K8S] - executor fails to be restarted after it goes to

[jira] [Resolved] (SPARK-32962) Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-32962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32962. -- Resolution: Invalid Looks more like a question. Let's ask it to the mailing list to get some

[jira] [Resolved] (SPARK-32965) pyspark reading csv files with utf_16le encoding

[ https://issues.apache.org/jira/browse/SPARK-32965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32965. -- Resolution: Duplicate > pyspark reading csv files with utf_16le encoding >

[jira] [Updated] (SPARK-32961) PySpark CSV read with UTF-16 encoding is not working correctly

[ https://issues.apache.org/jira/browse/SPARK-32961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bui Bao Anh updated SPARK-32961: Attachment: pyspark utf-16le.png > PySpark CSV read with UTF-16 encoding is not working correctly

[jira] [Updated] (SPARK-32961) PySpark CSV read with UTF-16 encoding is not working correctly

[ https://issues.apache.org/jira/browse/SPARK-32961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bui Bao Anh updated SPARK-32961: Attachment: pyspark utf-16 with multiline csv.png > PySpark CSV read with UTF-16 encoding is not

[jira] [Comment Edited] (SPARK-32961) PySpark CSV read with UTF-16 encoding is not working correctly

[ https://issues.apache.org/jira/browse/SPARK-32961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201356#comment-17201356 ] Hyukjin Kwon edited comment on SPARK-32961 at 9/24/20, 8:11 AM: For

[jira] [Resolved] (SPARK-32961) PySpark CSV read with UTF-16 encoding is not working correctly

[ https://issues.apache.org/jira/browse/SPARK-32961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32961. -- Resolution: Won't Fix > PySpark CSV read with UTF-16 encoding is not working correctly >

[jira] [Commented] (SPARK-32961) PySpark CSV read with UTF-16 encoding is not working correctly

[ https://issues.apache.org/jira/browse/SPARK-32961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201363#comment-17201363 ] Hyukjin Kwon commented on SPARK-32961: -- For the issue itself, I am almost 100% sure we can't fix

[jira] [Created] (SPARK-32986) Add bucket scan info in explain output of FileSourceScanExec

Cheng Su created SPARK-32986: Summary: Add bucket scan info in explain output of FileSourceScanExec Key: SPARK-32986 URL: https://issues.apache.org/jira/browse/SPARK-32986 Project: Spark Issue

[jira] [Commented] (SPARK-32956) Duplicate Columns in a csv file

[ https://issues.apache.org/jira/browse/SPARK-32956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201331#comment-17201331 ] Hyukjin Kwon commented on SPARK-32956: -- So what's the issue here? It can read the duplicated

[jira] [Commented] (SPARK-32961) PySpark CSV read with UTF-16 encoding is not working correctly

[ https://issues.apache.org/jira/browse/SPARK-32961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201356#comment-17201356 ] Hyukjin Kwon commented on SPARK-32961: -- For UTF-16LE or UTF-16BE, the file {{sendo.csv}} has to be

[jira] [Commented] (SPARK-32961) PySpark CSV read with UTF-16 encoding is not working correctly

[ https://issues.apache.org/jira/browse/SPARK-32961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201338#comment-17201338 ] Hyukjin Kwon commented on SPARK-32961: -- UTF-16 doesn't correctly work with CSV when {{multiLine}}

[jira] [Commented] (SPARK-32983) Spark SQL INTERSECT ALL does not keep all rows.

[ https://issues.apache.org/jira/browse/SPARK-32983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201383#comment-17201383 ] Peter Toth commented on SPARK-32983: [~willddy], I think the 1 row result is correct as according to

[jira] [Commented] (SPARK-32924) Web UI sort on duration is wrong

[ https://issues.apache.org/jira/browse/SPARK-32924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201329#comment-17201329 ] Hyukjin Kwon commented on SPARK-32924: -- Which codes did you run? > Web UI sort on duration is

[jira] [Commented] (SPARK-32961) PySpark CSV read with UTF-16 encoding is not working correctly

[ https://issues.apache.org/jira/browse/SPARK-32961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201351#comment-17201351 ] Bui Bao Anh commented on SPARK-32961: - Thanks a lot [~hyukjin.kwon],  I tried and it works with

[jira] [Resolved] (SPARK-32983) Spark SQL INTERSECT ALL does not keep all rows.

[ https://issues.apache.org/jira/browse/SPARK-32983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Will Du resolved SPARK-32983. - Resolution: Not A Problem [~petertoth]. You are correct. I close this issue as not a problem. > Spark

[jira] [Commented] (SPARK-32978) Incorrect number of dynamic part metric

[ https://issues.apache.org/jira/browse/SPARK-32978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201536#comment-17201536 ] Yuming Wang commented on SPARK-32978: - [~Chen Zhang] It is because your written files is also 50.

[jira] [Commented] (SPARK-32893) Structured Streaming and Dynamic Allocation on StandaloneCluster

[ https://issues.apache.org/jira/browse/SPARK-32893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201527#comment-17201527 ] Duarte Ferreira commented on SPARK-32893: - Further testing has shown that it happens even if the

[jira] [Created] (SPARK-32990) Migrate REFRESH TABLE to new resolution framework

Terry Kim created SPARK-32990: - Summary: Migrate REFRESH TABLE to new resolution framework Key: SPARK-32990 URL: https://issues.apache.org/jira/browse/SPARK-32990 Project: Spark Issue Type:

[jira] [Created] (SPARK-32993) pyspark on yarn executor hang after some time

zhao yufei created SPARK-32993: -- Summary: pyspark on yarn executor hang after some time Key: SPARK-32993 URL: https://issues.apache.org/jira/browse/SPARK-32993 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-32991) RESET can clear StaticSQLConfs

Herman van Hövell created SPARK-32991: - Summary: RESET can clear StaticSQLConfs Key: SPARK-32991 URL: https://issues.apache.org/jira/browse/SPARK-32991 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-32381) Expose the ability for users to use parallel file & avoid location information discovery in RDDs

[ https://issues.apache.org/jira/browse/SPARK-32381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-32381. -- Fix Version/s: 3.1.0 Resolution: Fixed > Expose the ability for users to use parallel

[jira] [Assigned] (SPARK-32381) Expose the ability for users to use parallel file & avoid location information discovery in RDDs

[ https://issues.apache.org/jira/browse/SPARK-32381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau reassigned SPARK-32381: Assignee: Chao Sun > Expose the ability for users to use parallel file & avoid location

[jira] [Updated] (SPARK-32992) In OracleDialect, "RowID" SQL type should be converted into "String" Catalyst type

[ https://issues.apache.org/jira/browse/SPARK-32992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peng Cheng updated SPARK-32992: --- Description: Most JDBC drivers use long SQL type for dataset row ID:   (in

[jira] [Updated] (SPARK-32992) In OracleDialect, "RowID" SQL type should be converted into "String" Catalyst type

[ https://issues.apache.org/jira/browse/SPARK-32992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peng Cheng updated SPARK-32992: --- Description: Most JDBC drivers use long SQL type for dataset row ID:   (in

[jira] [Updated] (SPARK-32988) Spark 2.3 and 2.4 backward compatibility problems due to ExternalCatalog

[ https://issues.apache.org/jira/browse/SPARK-32988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dinesh updated SPARK-32988: --- Summary: Spark 2.3 and 2.4 backward compatibility problems due to ExternalCatalog (was: Spark 2.3 and 2.4

[jira] [Updated] (SPARK-32988) ExternalCatalog backward compatibility problems due to ExternalCatalog

[ https://issues.apache.org/jira/browse/SPARK-32988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dinesh updated SPARK-32988: --- Summary: ExternalCatalog backward compatibility problems due to ExternalCatalog (was: Spark 2.3 and 2.4

[jira] [Commented] (SPARK-32889) orc table column name doesn't support special characters.

[ https://issues.apache.org/jira/browse/SPARK-32889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201780#comment-17201780 ] Apache Spark commented on SPARK-32889: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Commented] (SPARK-32991) RESET can clear StaticSQLConfs

[ https://issues.apache.org/jira/browse/SPARK-32991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201806#comment-17201806 ] Xiao Li commented on SPARK-32991: - ping [~Qin Yao] [~yumwang] > RESET can clear StaticSQLConfs >

[jira] [Commented] (SPARK-32978) Incorrect number of dynamic part metric

[ https://issues.apache.org/jira/browse/SPARK-32978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201516#comment-17201516 ] Chen Zhang commented on SPARK-32978: Hello, [~yumwang] I used the default config Spark to run this

[jira] [Comment Edited] (SPARK-32893) Structured Streaming and Dynamic Allocation on StandaloneCluster

[ https://issues.apache.org/jira/browse/SPARK-32893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201527#comment-17201527 ] Duarte Ferreira edited comment on SPARK-32893 at 9/24/20, 1:39 PM: ---

[jira] [Resolved] (SPARK-32954) Add jakarta.servlet-api test dependency to yarn module to avoid classpath badcase of UTs

[ https://issues.apache.org/jira/browse/SPARK-32954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32954. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29824

[jira] [Assigned] (SPARK-32954) Add jakarta.servlet-api test dependency to yarn module to avoid classpath badcase of UTs

[ https://issues.apache.org/jira/browse/SPARK-32954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-32954: - Assignee: Yang Jie > Add jakarta.servlet-api test dependency to yarn module to avoid

[jira] [Assigned] (SPARK-32987) Pass all `mesos` module UTs in Scala 2.13

[ https://issues.apache.org/jira/browse/SPARK-32987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-32987: - Assignee: Yang Jie > Pass all `mesos` module UTs in Scala 2.13 >

[jira] [Resolved] (SPARK-32987) Pass all `mesos` module UTs in Scala 2.13

[ https://issues.apache.org/jira/browse/SPARK-32987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32987. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29865

[jira] [Commented] (SPARK-32153) .m2 repository corruption happens

[ https://issues.apache.org/jira/browse/SPARK-32153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201679#comment-17201679 ] Shane Knapp commented on SPARK-32153: - i wiped all of the .m2 dirs on the centos workers.  seems