This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch
dependabot/cargo/native/main/object_store-0.13.0
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
omit 27d8ed6eb chore(deps): bump object_store from 0.12.4 to 0.13.0 in
/native
add f63c6a6aa feat: implement framework to support multiple pyspark
benchmarks (#3080)
add 267ad4c56 feat: add support for datediff expression (#3145)
add e4a014290 feat: Add support for `unix_timestamp` function (#2936)
add 0041bc5a3 feat: add support for last_day expression (#3143)
add d9ea22b1d docs: Add common pitfalls and improve PR checklist in
development guide (#3231)
add ea2662904 chore: add script to regenerate golden files for plan
stability tests (#3204)
add 313a9d94f Fix clippy warnings for Rust 1.93 (#3239)
add 5e9a1ca07 perf: reduce nativeIcebergScanMetadata serialization points
(#3243)
add 48776fedf perf: reduce GC pressure in protobuf serialization (#3242)
add 035aeffe6 feat: Support left expression (#3206)
add 5d6ed6115 perf: cache serialized query plans to avoid per-partition
serialization (#3246)
add c6c300241 build: build native library once and share across CI test
jobs (#3249)
add f538424d3 Experimental: Native CSV files read (#3044)
add ea264a39a fix: add missing datafusion-datasource dependency (#3252)
add 3a370093b chore(deps): bump object_store from 0.12.4 to 0.13.0 in
/native
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (27d8ed6eb)
\
N -- N -- N
refs/heads/dependabot/cargo/native/main/object_store-0.13.0 (3a370093b)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/actions/java-test/action.yaml | 5 +
.github/actions/setup-spark-builder/action.yaml | 14 +-
.github/workflows/pr_build_linux.yml | 88 +++++++-
.github/workflows/pr_build_macos.yml | 68 +++++-
.github/workflows/spark_sql_test.yml | 60 +++++
.gitignore | 1 +
benchmarks/pyspark/README.md | 111 ++++++++--
benchmarks/pyspark/benchmarks/__init__.py | 79 +++++++
benchmarks/pyspark/benchmarks/base.py | 127 +++++++++++
benchmarks/pyspark/benchmarks/shuffle.py | 130 +++++++++++
benchmarks/pyspark/run_benchmark.py | 109 +++++-----
.../main/scala/org/apache/comet/CometConf.scala | 10 +
dev/benchmarks/comet-tpch.sh | 3 +-
dev/benchmarks/spark-tpch.sh | 3 +-
dev/benchmarks/tpcbench.py | 13 +-
dev/regenerate-golden-files.sh | 178 +++++++++++++++
docs/source/contributor-guide/development.md | 137 +++++++++++-
docs/source/user-guide/latest/configs.md | 5 +
docs/spark_expressions_support.md | 2 +-
native/Cargo.lock | 78 +++++++
native/Cargo.toml | 8 +
native/core/Cargo.toml | 1 +
native/core/src/execution/expressions/mod.rs | 1 +
native/core/src/execution/expressions/temporal.rs | 162 ++++++++++++++
native/core/src/execution/jni_api.rs | 12 +-
native/core/src/execution/operators/csv_scan.rs | 86 ++++++++
native/core/src/execution/operators/mod.rs | 2 +
native/core/src/execution/planner.rs | 105 ++++-----
.../src/execution/planner/expression_registry.rs | 27 ++-
.../src/execution/planner/operator_registry.rs | 2 +
native/core/src/parquet/schema_adapter.rs | 5 +-
native/proto/src/proto/expr.proto | 8 +-
native/proto/src/proto/operator.proto | 20 ++
native/spark-expr/src/comet_scalar_funcs.rs | 5 +-
.../datetime_funcs/{date_trunc.rs => date_diff.rs} | 67 +++---
native/spark-expr/src/datetime_funcs/mod.rs | 4 +
.../src/datetime_funcs/unix_timestamp.rs | 242 +++++++++++++++++++++
native/spark-expr/src/lib.rs | 5 +-
.../org/apache/comet/parquet/ParquetFilters.scala | 13 +-
.../org/apache/comet/rules/CometExecRule.scala | 6 +-
.../org/apache/comet/rules/CometScanRule.scala | 42 ++++
.../org/apache/comet/serde/QueryPlanSerde.scala | 4 +
.../scala/org/apache/comet/serde/datetime.scala | 60 ++++-
.../comet/serde/operator/CometNativeScan.scala | 52 +----
.../org/apache/comet/serde/operator/package.scala | 70 ++++++
.../scala/org/apache/comet/serde/strings.scala | 32 ++-
.../spark/sql/comet/CometBatchScanExec.scala | 15 +-
.../spark/sql/comet/CometCsvNativeScanExec.scala | 142 ++++++++++++
.../apache/spark/sql/comet/CometExecUtils.scala | 6 +-
.../sql/comet/CometIcebergNativeScanExec.scala | 6 +-
.../spark/sql/comet/CometNativeWriteExec.scala | 23 +-
.../sql/comet/CometTakeOrderedAndProjectExec.scala | 29 ++-
.../org/apache/spark/sql/comet/operators.scala | 61 ++++--
.../test-data/{csv-test-1.csv => csv-test-2.csv} | 1 +
.../org/apache/comet/CometExpressionSuite.scala | 46 ++++
.../comet/CometTemporalExpressionSuite.scala | 136 +++++++++++-
.../apache/comet/csv/CometCsvNativeReadSuite.scala | 89 ++++++++
.../CometDatetimeExpressionBenchmark.scala | 49 ++++-
58 files changed, 2565 insertions(+), 300 deletions(-)
create mode 100644 benchmarks/pyspark/benchmarks/__init__.py
create mode 100644 benchmarks/pyspark/benchmarks/base.py
create mode 100644 benchmarks/pyspark/benchmarks/shuffle.py
create mode 100755 dev/regenerate-golden-files.sh
create mode 100644 native/core/src/execution/expressions/temporal.rs
create mode 100644 native/core/src/execution/operators/csv_scan.rs
copy native/spark-expr/src/datetime_funcs/{date_trunc.rs => date_diff.rs} (52%)
create mode 100644 native/spark-expr/src/datetime_funcs/unix_timestamp.rs
create mode 100644
spark/src/main/scala/org/apache/comet/serde/operator/package.scala
create mode 100644
spark/src/main/scala/org/apache/spark/sql/comet/CometCsvNativeScanExec.scala
copy spark/src/test/resources/test-data/{csv-test-1.csv => csv-test-2.csv}
(52%)
create mode 100644
spark/src/test/scala/org/apache/comet/csv/CometCsvNativeReadSuite.scala
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
