[jira] [Created] (SPARK-34075) Hidden directories are being listed for partition inference

2021-01-11 Thread Burak Yavuz (Jira)
Burak Yavuz created SPARK-34075: --- Summary: Hidden directories are being listed for partition inference Key: SPARK-34075 URL: https://issues.apache.org/jira/browse/SPARK-34075 Project: Spark

[jira] [Resolved] (SPARK-31255) DataSourceV2: Add metadata columns

2020-11-18 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-31255. - Fix Version/s: 3.1.0 Assignee: Ryan Blue Resolution: Done Resolved by 

[jira] [Commented] (SPARK-31640) Support SHOW PARTITIONS for DataSource V2 tables

2020-05-08 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102746#comment-17102746 ] Burak Yavuz commented on SPARK-31640: - Hi [~younggyuchun],   I'd take a look at how SHOW

[jira] [Created] (SPARK-31640) Support SHOW PARTITIONS for DataSource V2 tables

2020-05-04 Thread Burak Yavuz (Jira)
Burak Yavuz created SPARK-31640: --- Summary: Support SHOW PARTITIONS for DataSource V2 tables Key: SPARK-31640 URL: https://issues.apache.org/jira/browse/SPARK-31640 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-31624) SHOW TBLPROPERTIES doesn't handle Session Catalog correctly

2020-05-01 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-31624: --- Assignee: Burak Yavuz > SHOW TBLPROPERTIES doesn't handle Session Catalog correctly >

[jira] [Created] (SPARK-31624) SHOW TBLPROPERTIES doesn't handle Session Catalog correctly

2020-05-01 Thread Burak Yavuz (Jira)
Burak Yavuz created SPARK-31624: --- Summary: SHOW TBLPROPERTIES doesn't handle Session Catalog correctly Key: SPARK-31624 URL: https://issues.apache.org/jira/browse/SPARK-31624 Project: Spark

[jira] [Resolved] (SPARK-29314) ProgressReporter.extractStateOperatorMetrics should not overwrite updated as 0 when it actually runs a batch even with no data

2020-04-08 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-29314. - Fix Version/s: 3.0.0 Resolution: Fixed Resolved by 

[jira] [Assigned] (SPARK-29314) ProgressReporter.extractStateOperatorMetrics should not overwrite updated as 0 when it actually runs a batch even with no data

2020-04-08 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-29314: --- Assignee: Jungtaek Lim > ProgressReporter.extractStateOperatorMetrics should not overwrite

[jira] [Resolved] (SPARK-31278) numOutputRows shows value from last micro batch when there is no new data

2020-04-07 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-31278. - Resolution: Fixed > numOutputRows shows value from last micro batch when there is no new data >

[jira] [Assigned] (SPARK-31278) numOutputRows shows value from last micro batch when there is no new data

2020-04-07 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-31278: --- Assignee: Burak Yavuz > numOutputRows shows value from last micro batch when there is no

[jira] [Commented] (SPARK-31278) numOutputRows shows value from last micro batch when there is no new data

2020-04-07 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077695#comment-17077695 ] Burak Yavuz commented on SPARK-31278: - Resolved by [https://github.com/apache/spark/pull/28040] >

[jira] [Updated] (SPARK-31278) numOutputRows shows value from last micro batch when there is no new data

2020-04-07 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-31278: Fix Version/s: 3.0.0 > numOutputRows shows value from last micro batch when there is no new data

[jira] [Created] (SPARK-31278) numOutputRows shows value from last micro batch when there is no new data

2020-03-26 Thread Burak Yavuz (Jira)
Burak Yavuz created SPARK-31278: --- Summary: numOutputRows shows value from last micro batch when there is no new data Key: SPARK-31278 URL: https://issues.apache.org/jira/browse/SPARK-31278 Project:

[jira] [Resolved] (SPARK-31178) sql("INSERT INTO v2DataSource ...").collect() double inserts

2020-03-18 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-31178. - Fix Version/s: 3.0.0 Assignee: Burak Yavuz Resolution: Fixed Resolved by 

[jira] [Commented] (SPARK-31178) sql("INSERT INTO v2DataSource ...").collect() double inserts

2020-03-17 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061292#comment-17061292 ] Burak Yavuz commented on SPARK-31178: - cc [~wenchen] [~rdblue] > sql("INSERT INTO v2DataSource

[jira] [Created] (SPARK-31178) sql("INSERT INTO v2DataSource ...").collect() double inserts

2020-03-17 Thread Burak Yavuz (Jira)
Burak Yavuz created SPARK-31178: --- Summary: sql("INSERT INTO v2DataSource ...").collect() double inserts Key: SPARK-31178 URL: https://issues.apache.org/jira/browse/SPARK-31178 Project: Spark

[jira] [Created] (SPARK-31061) Impossible to change the provider of a table in the HiveMetaStore

2020-03-05 Thread Burak Yavuz (Jira)
Burak Yavuz created SPARK-31061: --- Summary: Impossible to change the provider of a table in the HiveMetaStore Key: SPARK-31061 URL: https://issues.apache.org/jira/browse/SPARK-31061 Project: Spark

[jira] [Created] (SPARK-30924) Add additional validation into Merge Into

2020-02-22 Thread Burak Yavuz (Jira)
Burak Yavuz created SPARK-30924: --- Summary: Add additional validation into Merge Into Key: SPARK-30924 URL: https://issues.apache.org/jira/browse/SPARK-30924 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-29908) Support partitioning for DataSource V2 tables in DataFrameWriter.save

2020-02-19 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-29908. - Resolution: Duplicate > Support partitioning for DataSource V2 tables in DataFrameWriter.save >

[jira] [Commented] (SPARK-30814) Add Columns references should be able to resolve each other

2020-02-13 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036418#comment-17036418 ] Burak Yavuz commented on SPARK-30814: - cc [~cloud_fan] [~imback82], can we prioritize this over

[jira] [Created] (SPARK-30814) Add Columns references should be able to resolve each other

2020-02-13 Thread Burak Yavuz (Jira)
Burak Yavuz created SPARK-30814: --- Summary: Add Columns references should be able to resolve each other Key: SPARK-30814 URL: https://issues.apache.org/jira/browse/SPARK-30814 Project: Spark

[jira] [Assigned] (SPARK-30697) Handle database and namespace exceptions in catalog.isView

2020-02-02 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-30697: --- Assignee: Burak Yavuz > Handle database and namespace exceptions in catalog.isView >

[jira] [Resolved] (SPARK-30697) Handle database and namespace exceptions in catalog.isView

2020-02-02 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-30697. - Fix Version/s: 3.0.0 Resolution: Fixed Resolved by 

[jira] [Created] (SPARK-30697) Handle database and namespace exceptions in catalog.isView

2020-01-31 Thread Burak Yavuz (Jira)
Burak Yavuz created SPARK-30697: --- Summary: Handle database and namespace exceptions in catalog.isView Key: SPARK-30697 URL: https://issues.apache.org/jira/browse/SPARK-30697 Project: Spark

[jira] [Resolved] (SPARK-30669) Introduce AdmissionControl API to Structured Streaming

2020-01-30 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-30669. - Fix Version/s: 3.0.0 Assignee: Burak Yavuz Resolution: Done Resolved by 

[jira] [Created] (SPARK-30669) Introduce AdmissionControl API to Structured Streaming

2020-01-28 Thread Burak Yavuz (Jira)
Burak Yavuz created SPARK-30669: --- Summary: Introduce AdmissionControl API to Structured Streaming Key: SPARK-30669 URL: https://issues.apache.org/jira/browse/SPARK-30669 Project: Spark Issue

[jira] [Resolved] (SPARK-30314) Add identifier and catalog information to DataSourceV2Relation

2020-01-26 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-30314. - Fix Version/s: 3.0.0 Resolution: Fixed Resolved by 

[jira] [Assigned] (SPARK-30314) Add identifier and catalog information to DataSourceV2Relation

2020-01-26 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-30314: --- Assignee: Yuchen Huo > Add identifier and catalog information to DataSourceV2Relation >

[jira] [Commented] (SPARK-30612) can't resolve qualified column name with v2 tables

2020-01-23 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022678#comment-17022678 ] Burak Yavuz commented on SPARK-30612: - I prefer SubqueryAlias. We need to support all degrees of the

[jira] [Commented] (SPARK-30615) normalize the column name in AlterTable

2020-01-23 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022577#comment-17022577 ] Burak Yavuz commented on SPARK-30615: - I actually had a PR in progress on this. Let me push that >

[jira] [Commented] (SPARK-30612) can't resolve qualified column name with v2 tables

2020-01-23 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022343#comment-17022343 ] Burak Yavuz commented on SPARK-30612: - SPARK-30314 should help make this work easier > can't

[jira] [Resolved] (SPARK-29219) DataSourceV2: Support all SaveModes in DataFrameWriter.save

2020-01-09 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-29219. - Fix Version/s: 3.0.0 Resolution: Done Resolved by 

[jira] [Assigned] (SPARK-29219) DataSourceV2: Support all SaveModes in DataFrameWriter.save

2020-01-09 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-29219: --- Assignee: Burak Yavuz > DataSourceV2: Support all SaveModes in DataFrameWriter.save >

[jira] [Updated] (SPARK-30334) Add metadata around semi-structured columns to Spark

2019-12-23 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-30334: Description: Semi-structured data is used widely in the data industry for reporting events in a

[jira] [Created] (SPARK-30334) Add metadata around semi-structured columns to Spark

2019-12-23 Thread Burak Yavuz (Jira)
Burak Yavuz created SPARK-30334: --- Summary: Add metadata around semi-structured columns to Spark Key: SPARK-30334 URL: https://issues.apache.org/jira/browse/SPARK-30334 Project: Spark Issue

[jira] [Created] (SPARK-30324) Simplify API for JSON access in DataFrames/SQL

2019-12-20 Thread Burak Yavuz (Jira)
Burak Yavuz created SPARK-30324: --- Summary: Simplify API for JSON access in DataFrames/SQL Key: SPARK-30324 URL: https://issues.apache.org/jira/browse/SPARK-30324 Project: Spark Issue Type: New

[jira] [Resolved] (SPARK-30143) StreamingQuery.stop() should not block indefinitely

2019-12-13 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-30143. - Fix Version/s: 3.0.0 Resolution: Done Resolved as part of 

[jira] [Assigned] (SPARK-30143) StreamingQuery.stop() should not block indefinitely

2019-12-13 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-30143: --- Assignee: Burak Yavuz > StreamingQuery.stop() should not block indefinitely >

[jira] [Resolved] (SPARK-30241) Make the target table relations not children of DELETE/UPDATE/MERGE

2019-12-12 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-30241. - Resolution: Not A Problem > Make the target table relations not children of DELETE/UPDATE/MERGE

[jira] [Created] (SPARK-30241) Make the target table relations not children of DELETE/UPDATE/MERGE

2019-12-12 Thread Burak Yavuz (Jira)
Burak Yavuz created SPARK-30241: --- Summary: Make the target table relations not children of DELETE/UPDATE/MERGE Key: SPARK-30241 URL: https://issues.apache.org/jira/browse/SPARK-30241 Project: Spark

[jira] [Created] (SPARK-30143) StreamingQuery.stop() should not block indefinitely

2019-12-05 Thread Burak Yavuz (Jira)
Burak Yavuz created SPARK-30143: --- Summary: StreamingQuery.stop() should not block indefinitely Key: SPARK-30143 URL: https://issues.apache.org/jira/browse/SPARK-30143 Project: Spark Issue

[jira] [Commented] (SPARK-29900) make relation lookup behavior consistent within Spark SQL

2019-11-14 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974436#comment-16974436 ] Burak Yavuz commented on SPARK-29900: - I definitely agree the behavior is very confusing here. (For

[jira] [Created] (SPARK-29778) saveAsTable append mode is not passing writer options

2019-11-06 Thread Burak Yavuz (Jira)
Burak Yavuz created SPARK-29778: --- Summary: saveAsTable append mode is not passing writer options Key: SPARK-29778 URL: https://issues.apache.org/jira/browse/SPARK-29778 Project: Spark Issue

[jira] [Created] (SPARK-29568) Add flag to stop existing stream when new copy starts

2019-10-23 Thread Burak Yavuz (Jira)
Burak Yavuz created SPARK-29568: --- Summary: Add flag to stop existing stream when new copy starts Key: SPARK-29568 URL: https://issues.apache.org/jira/browse/SPARK-29568 Project: Spark Issue

[jira] [Resolved] (SPARK-29352) Move active streaming query state to the SharedState

2019-10-23 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-29352. - Fix Version/s: 3.0.0 Resolution: Fixed Resolved by 

[jira] [Assigned] (SPARK-29352) Move active streaming query state to the SharedState

2019-10-09 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-29352: --- Assignee: Burak Yavuz > Move active streaming query state to the SharedState >

[jira] [Created] (SPARK-29352) Move active streaming query state to the SharedState

2019-10-03 Thread Burak Yavuz (Jira)
Burak Yavuz created SPARK-29352: --- Summary: Move active streaming query state to the SharedState Key: SPARK-29352 URL: https://issues.apache.org/jira/browse/SPARK-29352 Project: Spark Issue

[jira] [Updated] (SPARK-29219) DataSourceV2: Support all SaveModes in DataFrameWriter.save

2019-09-23 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-29219: Description: We currently don't support all save modes in DataFrameWriter.save as the

[jira] [Created] (SPARK-29219) DataSourceV2: Support all SaveModes in DataFrameWriter.save

2019-09-23 Thread Burak Yavuz (Jira)
Burak Yavuz created SPARK-29219: --- Summary: DataSourceV2: Support all SaveModes in DataFrameWriter.save Key: SPARK-29219 URL: https://issues.apache.org/jira/browse/SPARK-29219 Project: Spark

[jira] [Commented] (SPARK-29197) Remove saveModeForDSV2 in DataFrameWriter

2019-09-22 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935498#comment-16935498 ] Burak Yavuz commented on SPARK-29197: - Hi Ido, Thanks for your interest. I already have a PR on it

[jira] [Created] (SPARK-29197) Remove saveModeForDSV2 in DataFrameWriter

2019-09-20 Thread Burak Yavuz (Jira)
Burak Yavuz created SPARK-29197: --- Summary: Remove saveModeForDSV2 in DataFrameWriter Key: SPARK-29197 URL: https://issues.apache.org/jira/browse/SPARK-29197 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-28612) DataSourceV2: Add new DataFrameWriter API for v2

2019-09-19 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-28612. - Fix Version/s: 3.0.0 Assignee: Ryan Blue Resolution: Done Remerged in 

[jira] [Resolved] (SPARK-29030) Simplify lookupV2Relation

2019-09-18 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-29030. - Fix Version/s: 3.0.0 Assignee: John Zhuge Resolution: Done Resolved by

[jira] [Created] (SPARK-29127) Support partitioning for DataSource V2 tables in DataFrameWriter.save

2019-09-17 Thread Burak Yavuz (Jira)
Burak Yavuz created SPARK-29127: --- Summary: Support partitioning for DataSource V2 tables in DataFrameWriter.save Key: SPARK-29127 URL: https://issues.apache.org/jira/browse/SPARK-29127 Project: Spark

[jira] [Created] (SPARK-29062) Add V1_BATCH_WRITE to the TableCapabilityChecks in the Analyzer

2019-09-11 Thread Burak Yavuz (Jira)
Burak Yavuz created SPARK-29062: --- Summary: Add V1_BATCH_WRITE to the TableCapabilityChecks in the Analyzer Key: SPARK-29062 URL: https://issues.apache.org/jira/browse/SPARK-29062 Project: Spark

[jira] [Created] (SPARK-28964) saveAsTable doesn't pass in the data source information for V2 nodes

2019-09-03 Thread Burak Yavuz (Jira)
Burak Yavuz created SPARK-28964: --- Summary: saveAsTable doesn't pass in the data source information for V2 nodes Key: SPARK-28964 URL: https://issues.apache.org/jira/browse/SPARK-28964 Project: Spark

[jira] [Resolved] (SPARK-28628) Support namespaces in V2SessionCatalog

2019-09-03 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-28628. - Fix Version/s: 3.0.0 Resolution: Fixed > Support namespaces in V2SessionCatalog >

[jira] [Commented] (SPARK-28628) Support namespaces in V2SessionCatalog

2019-09-03 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921705#comment-16921705 ] Burak Yavuz commented on SPARK-28628: - Resolved by [https://github.com/apache/spark/pull/25363] >

[jira] [Assigned] (SPARK-28628) Support namespaces in V2SessionCatalog

2019-09-03 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-28628: --- Assignee: Ryan Blue > Support namespaces in V2SessionCatalog >

[jira] [Resolved] (SPARK-28612) DataSourceV2: Add new DataFrameWriter API for v2

2019-08-31 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-28612. - Fix Version/s: 3.0.0 Resolution: Fixed Resolved by 

[jira] [Assigned] (SPARK-28612) DataSourceV2: Add new DataFrameWriter API for v2

2019-08-31 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-28612: --- Assignee: Ryan Blue > DataSourceV2: Add new DataFrameWriter API for v2 >

[jira] [Created] (SPARK-28863) Add an AlreadyPlanned logical node that skips query planning

2019-08-23 Thread Burak Yavuz (Jira)
Burak Yavuz created SPARK-28863: --- Summary: Add an AlreadyPlanned logical node that skips query planning Key: SPARK-28863 URL: https://issues.apache.org/jira/browse/SPARK-28863 Project: Spark

[jira] [Commented] (SPARK-28635) create CatalogManager to track registered v2 catalogs

2019-08-21 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912621#comment-16912621 ] Burak Yavuz commented on SPARK-28635: - Also followed up by 

[jira] [Assigned] (SPARK-28565) DataSourceV2: DataFrameWriter.saveAsTable

2019-08-08 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-28565: --- Assignee: Burak Yavuz (was: John Zhuge) > DataSourceV2: DataFrameWriter.saveAsTable >

[jira] [Resolved] (SPARK-28565) DataSourceV2: DataFrameWriter.saveAsTable

2019-08-08 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-28565. - Resolution: Fixed Resolved by [https://github.com/apache/spark/pull/25330] > DataSourceV2:

[jira] [Created] (SPARK-28668) Support the V2SessionCatalog with AlterTable commands

2019-08-08 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-28668: --- Summary: Support the V2SessionCatalog with AlterTable commands Key: SPARK-28668 URL: https://issues.apache.org/jira/browse/SPARK-28668 Project: Spark Issue

[jira] [Created] (SPARK-28667) Support the V2SessionCatalog in insertInto

2019-08-08 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-28667: --- Summary: Support the V2SessionCatalog in insertInto Key: SPARK-28667 URL: https://issues.apache.org/jira/browse/SPARK-28667 Project: Spark Issue Type: Planned

[jira] [Created] (SPARK-28666) Support the V2SessionCatalog in saveAsTable

2019-08-08 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-28666: --- Summary: Support the V2SessionCatalog in saveAsTable Key: SPARK-28666 URL: https://issues.apache.org/jira/browse/SPARK-28666 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-28331) Catalogs.load always throws CatalogNotFoundException on loading built-in catalogs

2019-08-07 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-28331. - Resolution: Fixed Fix Version/s: 3.0.0 Resolved with 

[jira] [Assigned] (SPARK-28331) Catalogs.load always throws CatalogNotFoundException on loading built-in catalogs

2019-08-07 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-28331: --- Assignee: Gengliang Wang > Catalogs.load always throws CatalogNotFoundException on loading

[jira] [Resolved] (SPARK-27661) Add SupportsNamespaces interface for v2 catalogs

2019-08-04 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-27661. - Resolution: Fixed Fix Version/s: 3.0.0 Resolved by 

[jira] [Assigned] (SPARK-27661) Add SupportsNamespaces interface for v2 catalogs

2019-08-04 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-27661: --- Assignee: Ryan Blue > Add SupportsNamespaces interface for v2 catalogs >

[jira] [Created] (SPARK-28572) Add simple analysis checks to the V2 Create Table code paths

2019-07-30 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-28572: --- Summary: Add simple analysis checks to the V2 Create Table code paths Key: SPARK-28572 URL: https://issues.apache.org/jira/browse/SPARK-28572 Project: Spark

[jira] [Resolved] (SPARK-27845) DataSourceV2: InsertTable

2019-07-25 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-27845. - Resolution: Done Fix Version/s: 3.0.0 > DataSourceV2: InsertTable >

[jira] [Assigned] (SPARK-27845) DataSourceV2: InsertTable

2019-07-25 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-27845: --- Assignee: John Zhuge > DataSourceV2: InsertTable > - > >

[jira] [Commented] (SPARK-27845) DataSourceV2: InsertTable

2019-07-25 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893162#comment-16893162 ] Burak Yavuz commented on SPARK-27845: - Resolved with [https://github.com/apache/spark/pull/24832] >

[jira] [Commented] (SPARK-25472) Structured Streaming query.stop() doesn't always stop gracefully

2018-09-20 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622828#comment-16622828 ] Burak Yavuz commented on SPARK-25472: - Resolved by https://github.com/apache/spark/pull/22478 >

[jira] [Resolved] (SPARK-25472) Structured Streaming query.stop() doesn't always stop gracefully

2018-09-20 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-25472. - Resolution: Fixed Fix Version/s: 2.5.0 > Structured Streaming query.stop() doesn't

[jira] [Assigned] (SPARK-25472) Structured Streaming query.stop() doesn't always stop gracefully

2018-09-19 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-25472: --- Assignee: Burak Yavuz > Structured Streaming query.stop() doesn't always stop gracefully >

[jira] [Created] (SPARK-25472) Structured Streaming query.stop() doesn't always stop gracefully

2018-09-19 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-25472: --- Summary: Structured Streaming query.stop() doesn't always stop gracefully Key: SPARK-25472 URL: https://issues.apache.org/jira/browse/SPARK-25472 Project: Spark

[jira] [Resolved] (SPARK-24525) Provide an option to limit MemorySink memory usage

2018-06-15 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-24525. - Resolution: Fixed Fix Version/s: 2.4.0 Resolved by 

[jira] [Assigned] (SPARK-24525) Provide an option to limit MemorySink memory usage

2018-06-15 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-24525: --- Assignee: Mukul Murthy > Provide an option to limit MemorySink memory usage >

[jira] [Created] (SPARK-23515) JsonProtocol.sparkEventToJson can OOM when jsonifying an event

2018-02-25 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-23515: --- Summary: JsonProtocol.sparkEventToJson can OOM when jsonifying an event Key: SPARK-23515 URL: https://issues.apache.org/jira/browse/SPARK-23515 Project: Spark

[jira] [Commented] (SPARK-23194) from_json in FAILFAST mode doesn't fail fast, instead it just returns nulls

2018-01-23 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336183#comment-16336183 ] Burak Yavuz commented on SPARK-23194: - cc [~cloud_fan] seems like you touched that code last in

[jira] [Created] (SPARK-23194) from_json in FAILFAST mode doesn't fail fast, instead it just returns nulls

2018-01-23 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-23194: --- Summary: from_json in FAILFAST mode doesn't fail fast, instead it just returns nulls Key: SPARK-23194 URL: https://issues.apache.org/jira/browse/SPARK-23194 Project:

[jira] [Commented] (SPARK-23173) from_json can produce nulls for fields which are marked as non-nullable

2018-01-22 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334226#comment-16334226 ] Burak Yavuz commented on SPARK-23173: - In terms of usability, I prefer 1. In terms of the viewpoint

[jira] [Commented] (SPARK-23135) Accumulators don't show up properly in the Stages page anymore

2018-01-17 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329293#comment-16329293 ] Burak Yavuz commented on SPARK-23135: - cc [~vanzin] > Accumulators don't show up properly in the

[jira] [Updated] (SPARK-23135) Accumulators don't show up properly in the Stages page anymore

2018-01-17 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-23135: Environment:       was: Didn't do a lot of digging but may be caused by:

[jira] [Updated] (SPARK-23135) Accumulators don't show up properly in the Stages page anymore

2018-01-17 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-23135: Attachment: webUIAccumulatorRegression.png > Accumulators don't show up properly in the Stages

[jira] [Updated] (SPARK-23135) Accumulators don't show up properly in the Stages page anymore

2018-01-17 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-23135: Description: Didn't do a lot of digging but may be caused by:

[jira] [Created] (SPARK-23135) Accumulators don't show up properly in the Stages page anymore

2018-01-17 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-23135: --- Summary: Accumulators don't show up properly in the Stages page anymore Key: SPARK-23135 URL: https://issues.apache.org/jira/browse/SPARK-23135 Project: Spark

[jira] [Created] (SPARK-23094) Json Readers choose wrong encoding when bad records are present and fail

2018-01-16 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-23094: --- Summary: Json Readers choose wrong encoding when bad records are present and fail Key: SPARK-23094 URL: https://issues.apache.org/jira/browse/SPARK-23094 Project:

[jira] [Created] (SPARK-23092) Migrate MemoryStream to DataSource V2

2018-01-16 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-23092: --- Summary: Migrate MemoryStream to DataSource V2 Key: SPARK-23092 URL: https://issues.apache.org/jira/browse/SPARK-23092 Project: Spark Issue Type: Task

[jira] [Assigned] (SPARK-20168) Enable kinesis to start stream from Initial position specified by a timestamp

2017-12-25 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-20168: --- Assignee: Yash Sharma > Enable kinesis to start stream from Initial position specified by a

[jira] [Updated] (SPARK-20168) Enable kinesis to start stream from Initial position specified by a timestamp

2017-12-25 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-20168: Fix Version/s: 2.3.0 > Enable kinesis to start stream from Initial position specified by a

[jira] [Resolved] (SPARK-20168) Enable kinesis to start stream from Initial position specified by a timestamp

2017-12-25 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-20168. - Resolution: Done Target Version/s: 2.3.0 Resolved with

[jira] [Created] (SPARK-22238) EnsureStatefulOpPartitioning shouldn't ask for the child RDD before planning is completed

2017-10-10 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-22238: --- Summary: EnsureStatefulOpPartitioning shouldn't ask for the child RDD before planning is completed Key: SPARK-22238 URL: https://issues.apache.org/jira/browse/SPARK-22238

[jira] [Resolved] (SPARK-21977) SinglePartition optimizations break certain Streaming Stateful Aggregation requirements

2017-09-20 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-21977. - Resolution: Fixed Fix Version/s: 2.3.0 > SinglePartition optimizations break certain

[jira] [Updated] (SPARK-21977) SinglePartition optimizations break certain Streaming Stateful Aggregation requirements

2017-09-11 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-21977: Description: This is a bit hard to explain as there are several issues here, I'll try my best.

[jira] [Updated] (SPARK-21977) SinglePartition optimizations break certain Streaming Stateful Aggregation requirements

2017-09-11 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-21977: Summary: SinglePartition optimizations break certain Streaming Stateful Aggregation requirements

  1   2   3   4   >