This is an automated email from the ASF dual-hosted git repository.
dependabot[bot] pushed a change to branch
dependabot/cargo/native/main/object_store-0.14.1
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
omit 86f50e9fd9 chore(deps): bump object_store from 0.13.2 to 0.14.1 in
/native
add 1981091602 feat: add spark.comet.shuffle.maxBufferBytes to cap native
shuffle writer memory (#4989)
add 5ef4bd7a72 fix: widen local table scan child nullability to match
native kernels (#4843)
add 7b00c40cd1 perf: optimize `cast_binary_to_string` binary-format styles
(up to 27x faster) (#4912)
add 81c2a84fe3 perf: optimize `parse_string_to_decimal` (30-40% faster)
(#4916)
add 7979527bee chore: use DF `array_repeat` and `array_compact` (#4741)
add e2f5df6d04 chor: remove legacy use.lazyMaterialization (#4998)
add c7f9a1f977 perf: optimize `date_parser` / cast string to date (up to
2x faster) (#4917)
add 52140f3930 perf: optimize cast_decimal128_to_utf8 in
datafusion-comet-spark-expr (#4924)
add 56405e031f perf: optimize `date_trunc` (>2x faster) (#4915)
add 6934ef85e4 perf: optimize `spark_cast_float64_to_utf8` (~40% faster)
(#4918)
add d075ab326e docs: document Spark version adoption and support-lifetime
policy (#4977)
add f06aa31b3e feat: support approx_count_distinct aggregate expression
(#4819)
add 0761e549a4 perf: dedupe Iceberg residuals and delete files in native
scan serde (#4982)
add bc53ad5c56 feat: Iceberg table format V3: native table decryption,
fall back for other V3 features (#4991)
add 34c598b116 chore(deps): bump object_store from 0.13.2 to 0.14.1 in
/native
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (86f50e9fd9)
\
N -- N -- N
refs/heads/dependabot/cargo/native/main/object_store-0.14.1 (34c598b116)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/actions/java-test/action.yaml | 14 +-
.github/workflows/pr_build_linux.yml | 1 +
.github/workflows/pr_build_macos.yml | 1 +
dev/diffs/iceberg/1.11.0.diff | 25 +
docs/source/about/versioning_policy.md | 61 +-
.../expression-audits/agg_funcs.md | 7 +
.../expression-audits/array_funcs.md | 6 +-
docs/source/contributor-guide/native_shuffle.md | 4 +-
docs/source/contributor-guide/roadmap.md | 6 +-
.../latest/compatibility/spark-versions.md | 4 +-
docs/source/user-guide/latest/expressions.md | 4 +-
docs/source/user-guide/latest/iceberg.md | 22 +-
native/Cargo.lock | 533 ++++---
native/Cargo.toml | 8 +-
native/core/src/execution/jni_api.rs | 7 +-
.../core/src/execution/operators/iceberg_scan.rs | 57 +
native/core/src/execution/planner.rs | 497 +++---
native/proto/src/proto/expr.proto | 8 +
native/proto/src/proto/operator.proto | 120 +-
native/shuffle/benches/shuffle_writer.rs | 1 +
native/shuffle/src/bin/shuffle_bench.rs | 9 +
native/shuffle/src/partitioners/multi_partition.rs | 15 +-
native/shuffle/src/shuffle_writer.rs | 177 ++-
native/spark-expr/Cargo.toml | 21 +-
native/spark-expr/benches/cast_binary_to_string.rs | 115 ++
.../spark-expr/benches/cast_decimal_to_string.rs | 83 +
native/spark-expr/benches/cast_float_to_string.rs | 101 ++
native/spark-expr/benches/cast_from_string.rs | 55 +-
.../{cast_numeric.rs => cast_string_to_date.rs} | 67 +-
native/spark-expr/benches/date_trunc_array_fmt.rs | 91 ++
native/spark-expr/src/agg_funcs/hll_plus_plus.rs | 656 ++++++++
.../src/agg_funcs/hll_plus_plus_const.rs | 1666 ++++++++++++++++++++
native/spark-expr/src/agg_funcs/mod.rs | 3 +
native/spark-expr/src/array_funcs/array_compact.rs | 165 --
native/spark-expr/src/array_funcs/mod.rs | 2 -
native/spark-expr/src/comet_scalar_funcs.rs | 7 +-
native/spark-expr/src/conversion_funcs/cast.rs | 257 ++-
native/spark-expr/src/conversion_funcs/numeric.rs | 228 ++-
native/spark-expr/src/conversion_funcs/string.rs | 387 +++--
native/spark-expr/src/hash_funcs/mod.rs | 1 +
native/spark-expr/src/hash_funcs/xxhash64.rs | 2 +-
native/spark-expr/src/kernels/temporal.rs | 279 ++--
native/spark-expr/src/math_funcs/internal/mod.rs | 1 +
.../src/math_funcs/internal/normalize_nan.rs | 2 +-
spark/pom.xml | 7 +
.../main/scala/org/apache/comet/CometConf.scala | 24 +-
.../apache/comet/iceberg/IcebergReflection.scala | 99 ++
.../org/apache/comet/rules/CometScanRule.scala | 86 +-
.../org/apache/comet/serde/QueryPlanSerde.scala | 6 +-
.../scala/org/apache/comet/serde/aggregates.scala | 67 +-
.../main/scala/org/apache/comet/serde/arrays.scala | 54 +-
.../serde/operator/CometIcebergNativeScan.scala | 526 ++++--
.../apache/comet/serde/operator/CometSink.scala | 10 +-
.../spark/sql/comet/CometLocalTableScanExec.scala | 17 +-
.../shuffle/CometNativeShuffleWriter.scala | 1 +
.../apache/comet/shims/Spark4xCometExprShim.scala | 28 +-
.../aggregate/approx_count_distinct.sql | 158 ++
.../approx_count_distinct_collation.sql} | 27 +-
.../approx_count_distinct_timezone.sql} | 31 +-
.../sql-tests/expressions/array/array_compact.sql | 113 +-
.../sql-tests/expressions/array/array_repeat.sql | 93 +-
.../apache/comet/CometArrayExpressionSuite.scala | 31 +
.../org/apache/comet/CometFuzzIcebergBase.scala | 2 +-
.../org/apache/comet/CometFuzzIcebergSuite.scala | 206 +--
.../org/apache/comet/CometIcebergNativeSuite.scala | 496 ++++++
.../org/apache/comet/CometIcebergTestBase.scala | 2 +-
.../org/apache/comet/CometMapExpressionSuite.scala | 14 +
.../comet/exec/CometNativeShuffleSuite.scala | 34 +
.../apache/comet/parquet/ParquetReadSuite.scala | 19 +-
.../apache/comet/rules/CometExecRuleSuite.scala | 48 +
.../CometAggregateExpressionBenchmark.scala | 18 +-
.../apache/comet/CometIcebergEncryptionSuite.scala | 261 +++
.../org/apache/comet/iceberg/CometTestKMS.scala | 63 +
73 files changed, 6613 insertions(+), 1704 deletions(-)
create mode 100644 native/spark-expr/benches/cast_binary_to_string.rs
create mode 100644 native/spark-expr/benches/cast_decimal_to_string.rs
create mode 100644 native/spark-expr/benches/cast_float_to_string.rs
copy native/spark-expr/benches/{cast_numeric.rs => cast_string_to_date.rs}
(51%)
create mode 100644 native/spark-expr/benches/date_trunc_array_fmt.rs
create mode 100644 native/spark-expr/src/agg_funcs/hll_plus_plus.rs
create mode 100644 native/spark-expr/src/agg_funcs/hll_plus_plus_const.rs
delete mode 100644 native/spark-expr/src/array_funcs/array_compact.rs
create mode 100644
spark/src/test/resources/sql-tests/expressions/aggregate/approx_count_distinct.sql
copy
spark/src/test/resources/sql-tests/expressions/{conditional/nullif_zeroif.sql
=> aggregate/approx_count_distinct_collation.sql} (52%)
copy spark/src/test/resources/sql-tests/expressions/{datetime/add_months.sql
=> aggregate/approx_count_distinct_timezone.sql} (57%)
create mode 100644
spark/src/test/spark-4.1/org/apache/comet/CometIcebergEncryptionSuite.scala
create mode 100644
spark/src/test/spark-4.1/org/apache/comet/iceberg/CometTestKMS.scala
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
