[jira] [Updated] (SPARK-29611) Sort Kafka metadata by the number of messages

2019-11-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29611: -- Affects Version/s: 2.4.4 > Sort Kafka metadata by the number of messages >

[jira] [Resolved] (SPARK-29611) Optimize the display of Kafka metadata and sort by the number of messages

2019-11-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29611. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26266

[jira] [Updated] (SPARK-29611) Sort Kafka metadata by the number of messages

2019-11-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29611: -- Summary: Sort Kafka metadata by the number of messages (was: Optimize the display of Kafka

[jira] [Assigned] (SPARK-29611) Optimize the display of Kafka metadata and sort by the number of messages

2019-11-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29611: - Assignee: dengziming > Optimize the display of Kafka metadata and sort by the number

[jira] [Updated] (SPARK-29721) Spark SQL reads unnecessary nested fields from Parquet after using explode

2019-11-01 Thread Kai Kang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Kang updated SPARK-29721: - Description: This is a follow up for SPARK-4502. SPARK-4502 correctly addressed column pruning for

[jira] [Updated] (SPARK-29721) Spark SQL reads unnecessary nested fields from Parquet after using explode

2019-11-01 Thread Kai Kang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Kang updated SPARK-29721: - Description: This is a follow up for SPARK-4502. SPARK-4502 correctly addressed column pruning for

[jira] [Updated] (SPARK-29721) Spark SQL reads unnecessary nested fields from Parquet after using explode

2019-11-01 Thread Kai Kang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Kang updated SPARK-29721: - Description: This is a follow up for SPARK-4502. SPARK-4502 correctly addressed column pruning for

[jira] [Updated] (SPARK-29721) Spark SQL reads unnecessary nested fields from Parquet after using explode

2019-11-01 Thread Kai Kang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Kang updated SPARK-29721: - Description: This is a follow up for SPARK-4502. SPARK-4502 correctly addressed column pruning for

[jira] [Updated] (SPARK-29721) Spark SQL reads unnecessary nested fields from Parquet after using explode

2019-11-01 Thread Kai Kang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Kang updated SPARK-29721: - Description: This is a follow up for SPARK-4502. SPARK-4502 correctly addressed column pruning for

[jira] [Updated] (SPARK-29721) Spark SQL reads unnecessary nested fields from Parquet after using explode

2019-11-01 Thread Kai Kang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Kang updated SPARK-29721: - Description: This is a follow up for SPARK-4502. SPARK-4502 correctly addressed column pruning for

[jira] [Updated] (SPARK-29721) Spark SQL reads unnecessary nested fields from Parquet after using explode

2019-11-01 Thread Kai Kang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Kang updated SPARK-29721: - Description: This is a follow up for SPARK-4502. SPARK-4502 correctly addressed column pruning for

[jira] [Updated] (SPARK-29721) Spark SQL reads unnecessary nested fields from Parquet after using explode

2019-11-01 Thread Kai Kang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Kang updated SPARK-29721: - Description: This is a follow up for SPARK-4502. SPARK-4502 correctly addressed column pruning for

[jira] [Updated] (SPARK-29721) Spark SQL reads unnecessary nested fields from Parquet after using explode

2019-11-01 Thread Kai Kang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Kang updated SPARK-29721: - Description: This is a follow up for SPARK-4502. SPARK-4502 correctly addressed column pruning for

[jira] [Updated] (SPARK-29721) Spark SQL reads unnecessary nested fields from Parquet after using explode

2019-11-01 Thread Kai Kang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Kang updated SPARK-29721: - Description: This is a follow up for SPARK-4502. SPARK-4502 correctly addressed column pruning for

[jira] [Updated] (SPARK-29721) Spark SQL reads unnecessary nested fields from Parquet after using explode

2019-11-01 Thread Kai Kang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Kang updated SPARK-29721: - Description: This is a follow up for SPARK-4502. SPARK-4502 correctly addressed column pruning for

[jira] [Updated] (SPARK-29721) Spark SQL reads unnecessary nested fields from Parquet after using explode

2019-11-01 Thread Kai Kang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Kang updated SPARK-29721: - Description: This is a follow up for SPARK-4502. SPARK-4502 correctly addressed column pruning for

[jira] [Created] (SPARK-29721) Spark SQL reads unnecessary nested fields from Parquet after using explode

2019-11-01 Thread Kai Kang (Jira)
Kai Kang created SPARK-29721: Summary: Spark SQL reads unnecessary nested fields from Parquet after using explode Key: SPARK-29721 URL: https://issues.apache.org/jira/browse/SPARK-29721 Project: Spark

[jira] [Commented] (SPARK-29682) Failure when resolving conflicting references in Join:

2019-11-01 Thread sandeshyapuram (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16965216#comment-16965216 ] sandeshyapuram commented on SPARK-29682: [~imback82]  & [~cloud_fan] Currently I've worked my

[jira] [Created] (SPARK-29720) Add linux condition to make ProcfsMetricsGetter more complete

2019-11-01 Thread ulysses you (Jira)
ulysses you created SPARK-29720: --- Summary: Add linux condition to make ProcfsMetricsGetter more complete Key: SPARK-29720 URL: https://issues.apache.org/jira/browse/SPARK-29720 Project: Spark

[jira] [Commented] (SPARK-29719) Converted Metastore relations (ORC, Parquet) wouldn't update InMemoryFileIndex

2019-11-01 Thread Yuming Wang (Jira)
ode:java} > val spark = SparkSession.builder() > .master("yarn") > .enableHiveSupport > .config("spark.sql.hive.caseSensitiveInferenceMode", "NEVER_INFER") > .getOrCreate() > val df1 = spark.table("my_table").filter("d

[jira] [Comment Edited] (SPARK-29691) Estimator fit method fails to copy params (in PySpark)

2019-11-01 Thread Huaxin Gao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16965128#comment-16965128 ] Huaxin Gao edited comment on SPARK-29691 at 11/1/19 10:33 PM: -- I checked

[jira] [Comment Edited] (SPARK-29691) Estimator fit method fails to copy params (in PySpark)

2019-11-01 Thread Huaxin Gao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16965128#comment-16965128 ] Huaxin Gao edited comment on SPARK-29691 at 11/1/19 10:32 PM: -- I checked

[jira] [Commented] (SPARK-29691) Estimator fit method fails to copy params (in PySpark)

2019-11-01 Thread Huaxin Gao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16965128#comment-16965128 ] Huaxin Gao commented on SPARK-29691: I checked the doc and implementation. The Estimator fits the

[jira] [Created] (SPARK-29719) Converted Metastore relations (ORC, Parquet) wouldn't update InMemoryFileIndex

2019-11-01 Thread Alexander Bessonov (Jira)
.config("spark.sql.hive.caseSensitiveInferenceMode", "NEVER_INFER") .getOrCreate() val df1 = spark.table("my_table").filter("date=20191101") // Do something with `df1` // External process writes to the partition val df2 = spark.table("my_table").filter("d

[jira] [Closed] (SPARK-18910) Can't use UDF that jar file in hdfs

2019-11-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-18910. - > Can't use UDF that jar file in hdfs > --- > >

[jira] [Closed] (SPARK-21697) NPE & ExceptionInInitializerError trying to load UDF from HDFS

2019-11-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-21697. - > NPE & ExceptionInInitializerError trying to load UDF from HDFS >

[jira] [Resolved] (SPARK-21697) NPE & ExceptionInInitializerError trying to load UDF from HDFS

2019-11-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-21697. --- Resolution: Duplicate > NPE & ExceptionInInitializerError trying to load UDF from HDFS >

[jira] [Reopened] (SPARK-21697) NPE & ExceptionInInitializerError trying to load UTF from HDFS

2019-11-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-21697: --- > NPE & ExceptionInInitializerError trying to load UTF from HDFS >

[jira] [Updated] (SPARK-21697) NPE & ExceptionInInitializerError trying to load UDF from HDFS

2019-11-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21697: -- Summary: NPE & ExceptionInInitializerError trying to load UDF from HDFS (was: NPE &

[jira] [Updated] (SPARK-21697) NPE & ExceptionInInitializerError trying to load UDF from HDFS

2019-11-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21697: -- Labels: (was: bulk-closed) > NPE & ExceptionInInitializerError trying to load UDF from HDFS

[jira] [Updated] (SPARK-25694) URL.setURLStreamHandlerFactory causing incompatible HttpURLConnection issue

2019-11-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25694: -- Affects Version/s: 3.0.0 2.4.4 > URL.setURLStreamHandlerFactory

[jira] [Commented] (SPARK-25694) URL.setURLStreamHandlerFactory causing incompatible HttpURLConnection issue

2019-11-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16965036#comment-16965036 ] Dongjoon Hyun commented on SPARK-25694: --- Although SPARK-12868 adds `setURLStreamHandlerFactory`

[jira] [Assigned] (SPARK-29452) Improve tootip information for storage tab

2019-11-01 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-29452: Assignee: Rakesh Raushan > Improve tootip information for storage tab >

[jira] [Updated] (SPARK-29651) Incorrect parsing of interval seconds fraction

2019-11-01 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-29651: Fix Version/s: 2.4.5 > Incorrect parsing of interval seconds fraction >

[jira] [Created] (SPARK-29718) Support PARTITION BY [RANGE|LIST|HASH] and PARTITION OF in CREATE TABLE

2019-11-01 Thread Takeshi Yamamuro (Jira)
Takeshi Yamamuro created SPARK-29718: Summary: Support PARTITION BY [RANGE|LIST|HASH] and PARTITION OF in CREATE TABLE Key: SPARK-29718 URL: https://issues.apache.org/jira/browse/SPARK-29718

[jira] [Created] (SPARK-29717) Support [CREATE|DROP] RULE - define a new plan rewrite rule

2019-11-01 Thread Takeshi Yamamuro (Jira)
Takeshi Yamamuro created SPARK-29717: Summary: Support [CREATE|DROP] RULE - define a new plan rewrite rule Key: SPARK-29717 URL: https://issues.apache.org/jira/browse/SPARK-29717 Project: Spark

[jira] [Created] (SPARK-29716) Support [CREATE|DROP] TYPE

2019-11-01 Thread Takeshi Yamamuro (Jira)
Takeshi Yamamuro created SPARK-29716: Summary: Support [CREATE|DROP] TYPE Key: SPARK-29716 URL: https://issues.apache.org/jira/browse/SPARK-29716 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-29715) Support SELECT statements in VALUES of INSERT INTO

2019-11-01 Thread Takeshi Yamamuro (Jira)
Takeshi Yamamuro created SPARK-29715: Summary: Support SELECT statements in VALUES of INSERT INTO Key: SPARK-29715 URL: https://issues.apache.org/jira/browse/SPARK-29715 Project: Spark

[jira] [Resolved] (SPARK-29452) Improve tootip information for storage tab

2019-11-01 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29452. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26226

[jira] [Created] (SPARK-29714) Add insert.sql

2019-11-01 Thread Takeshi Yamamuro (Jira)
Takeshi Yamamuro created SPARK-29714: Summary: Add insert.sql Key: SPARK-29714 URL: https://issues.apache.org/jira/browse/SPARK-29714 Project: Spark Issue Type: Sub-task

[jira] [Resolved] (SPARK-29109) Add window.sql - Part 3

2019-11-01 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-29109. -- Fix Version/s: 3.0.0 Assignee: Dylan Guedes Resolution: Fixed

[jira] [Created] (SPARK-29713) Support Interval Unit Abbreviations in Interval Literals

2019-11-01 Thread Kent Yao (Jira)
Kent Yao created SPARK-29713: Summary: Support Interval Unit Abbreviations in Interval Literals Key: SPARK-29713 URL: https://issues.apache.org/jira/browse/SPARK-29713 Project: Spark Issue Type:

[jira] [Created] (SPARK-29712) fromDayTimeString() does not take into account the left bound

2019-11-01 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-29712: -- Summary: fromDayTimeString() does not take into account the left bound Key: SPARK-29712 URL: https://issues.apache.org/jira/browse/SPARK-29712 Project: Spark

[jira] [Assigned] (SPARK-29643) ALTER TABLE (DROP PARTITION) should look up catalog/table like v2 commands

2019-11-01 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-29643: --- Assignee: Huaxin Gao > ALTER TABLE (DROP PARTITION) should look up catalog/table like v2

[jira] [Resolved] (SPARK-29643) ALTER TABLE (DROP PARTITION) should look up catalog/table like v2 commands

2019-11-01 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-29643. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26303

[jira] [Resolved] (SPARK-29486) CalendarInterval should have 3 fields: months, days and microseconds

2019-11-01 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-29486. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26134

[jira] [Assigned] (SPARK-29486) CalendarInterval should have 3 fields: months, days and microseconds

2019-11-01 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-29486: --- Assignee: Liu, Linhong > CalendarInterval should have 3 fields: months, days and

[jira] [Created] (SPARK-29711) Dynamic adjust spark sql class log level in beeline

2019-11-01 Thread deshanxiao (Jira)
deshanxiao created SPARK-29711: -- Summary: Dynamic adjust spark sql class log level in beeline Key: SPARK-29711 URL: https://issues.apache.org/jira/browse/SPARK-29711 Project: Spark Issue Type:

[jira] [Commented] (SPARK-29710) Seeing offsets not resetting even when reset policy is configured explicitly

2019-11-01 Thread Shyam (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964696#comment-16964696 ] Shyam commented on SPARK-29710: --- @Gabor Somogyi Can you please help me , what is wrong here  > Seeing

[jira] [Commented] (SPARK-29707) Make PartitionFilters and PushedFilters abbreviate configurable in metadata

2019-11-01 Thread Hu Fuwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964655#comment-16964655 ] Hu Fuwang commented on SPARK-29707: --- I am working on this. > Make PartitionFilters and PushedFilters

[jira] [Created] (SPARK-29710) Seeing offsets not resetting even when reset policy is configured explicitly

2019-11-01 Thread Shyam (Jira)
Shyam created SPARK-29710: - Summary: Seeing offsets not resetting even when reset policy is configured explicitly Key: SPARK-29710 URL: https://issues.apache.org/jira/browse/SPARK-29710 Project: Spark

[jira] [Created] (SPARK-29709) structured streaming The offset in the checkpoint is suddenly reset to the earliest

2019-11-01 Thread test (Jira)
test created SPARK-29709: Summary: structured streaming The offset in the checkpoint is suddenly reset to the earliest Key: SPARK-29709 URL: https://issues.apache.org/jira/browse/SPARK-29709 Project: Spark