This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
from 1d3149fc1b90 [MINOR][PYTHON] Reject empty memory strings with
ValueError
add a11fd253b5a3 [SPARK-60734][SS] Prune the columns that the scans of
Data Source V2 streaming sources read
No new revisions were added by this update.
Summary of changes:
.../src/main/resources/error/error-conditions.json | 12 +
.../sql/connector/catalog/TableCapability.java | 22 +-
.../read/SupportsPushDownRequiredColumns.java | 6 +
.../spark/sql/errors/QueryExecutionErrors.scala | 23 +
.../datasources/v2/DataSourceV2Relation.scala | 26 +-
.../org/apache/spark/sql/internal/SQLConf.scala | 17 +
.../sql/connector/catalog/InMemoryBaseTable.scala | 99 +-
.../datasources/v2/DataSourceV2Strategy.scala | 41 +-
.../datasources/v2/columnar/ColumnarTable.scala | 12 +-
.../streaming/checkpointing/OffsetSeq.scala | 16 +
.../streaming/continuous/ContinuousExecution.scala | 28 +-
.../streaming/runtime/MicroBatchExecution.scala | 25 +-
.../runtime/StreamingV2ScanPushDown.scala | 498 ++++++++
.../v2/columnar/ColumnarDataSourceSuite.scala | 25 +
.../execution/streaming/OffsetSeqLogSuite.scala | 26 +
.../StreamingV2ScanPushDownRealTimeSuite.scala | 159 +++
.../streaming/StreamingV2ScanPushDownSuite.scala | 1325 ++++++++++++++++++++
17 files changed, 2314 insertions(+), 46 deletions(-)
create mode 100644
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/StreamingV2ScanPushDown.scala
create mode 100644
sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingV2ScanPushDownRealTimeSuite.scala
create mode 100644
sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingV2ScanPushDownSuite.scala
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]