This is an automated email from the ASF dual-hosted git repository.
gabotechs pushed a change to branch gabotechs/port-iceberg-rust-history
in repository https://gitbox.apache.org/repos/asf/datafusion-iceberg.git
at 9b1a2e9 deps: upgrade to DataFusion 55.0 and Arrow/Parquet 59.2
(#2958)
This branch includes the following new commits:
new 0905609 Basic Integration with Datafusion (#324)
new 19e6c87 feat: Extract FileRead and FileWrite trait (#364)
new 0452002 chore(deps): Update datafusion requirement from 37.0.0 to
38.0.0 (#369)
new 93bc0c4 deps: Bump arrow related crates to 52 (#403)
new 4e421f6 feat: Add storage features for iceberg (#400)
new 500ae71 chore: Use nightly toolchain for check (#445)
new a2984bc reuse docker container to save compute resources (#428)
new 9cb8da4 chore: remove compose obsolete version (#452) (#454)
new 820677c fix: Hms test on macos should use correct arch (#461)
new 4fd5a21 chore: Enable new rust code format settings (#483)
new 73f8992 feat: podman support (#489)
new 17ca188 chore: Bump datafusion to 41 (#548)
new db2301c Bump to version 0.3.0 (#549)
new 4e5195b test: refactor datafusion test with memory catalog (#557)
new 9468aa7 feat: support projection pushdown for datafusion iceberg
(#594)
new 15a59b3 Migrate to arrow-* v53 (#626)
new 85cdbc4 feat (datafusion integration): convert datafusion expr
filters to Iceberg Predicate (#588)
new e95278d making table provider pub (#650)
new a86ecaa test (datafusion): add test for table provider creation (#651)
new 430ed4d feat(datafusion): Support pushdown more datafusion exprs to
Iceberg (#649)
new 563d76c feat: implement IcebergTableProviderFactory for datafusion
(#600)
new 9c98c89 Upgrade to DataFusion 43 (#691)
new 0c88d1f datafusion: Create table provider for a snapshot. (#707)
new 8ee8e7a Fix error running data fusion queries - Physical input schema
should be the same as the one converted from logical input schema (#664)
new acfb0a8 Derive Clone for IcebergTableProvider (#722)
new b847bcd chore: Generate Changelog Dependencies for 0.4.0 release
(#812)
new 2ae61b2 Align schemas for DataFusion plan and stream (#829)
new 1b8972a chore: datafusion 44 upgrade (#867)
new 5a8583f ci: check MSRV correctly (#849)
new 8f8f58b Datafusion: Support cast operations (#821)
new d86825f Add pyspark DataFusion integration test (#850)
new c08b3ec fix: Do not extract expression from cast to date (#977)
new 0d41351 feat: nan_value_counts support (#907)
new ec439df refactor: Bump MSRV to 1.84 for preparing next release (#1185)
new ae721fc ci: Use taplo to replace cargo-sort (#1186)
new cbd9732 refactor: use the same MSRV for datafusion integration (#1197)
new 9f5ad68 chore: bump up arrow/parquet/datafusion (#1294)
new c5ea4c2 Expose datafusion table provider as python binding (#1324)
new 2dc29af Bump iceberg-rust version to 0.5.0 (Round 1) (#1342)
new 1a5b05d Bump iceberg-rust version to 0.5.0 (#1345)
new 60b46d0 Regenerate the licenses and update the allowed list (#1363)
new 285fcab Bump iceberg-rust version to 0.5.1 (#1380)
new 3a5c028 feat(datafusion): support metadata tables for Datafusion
(#879)
new 8ccaf10 refactor: Move memory catalog into core (#1480)
new 0ebed78 Preparing 0.6.0 release (update dep list) (#1525)
new d7827d7 feat(iceberg-datafusion): make IcebergTableScan public (#1570)
new 9da5fa8 feat(datafusion): Add IcebergCommitExec to commit the written
data files (#1588)
new c1e2bdd feat(datafusion): Implement IcebergWriteExec for DataFusion
write support (#1585)
new 7d2425f feat(arrow): Allow ArrowArrayAccessor to use field.name to
match fields (#1561)
new 26c0bec Include license and notice files in published crates (#1601)
new 7d9ccb1 feat(catalog): Implement catalog loader for in memory (#1623)
new 5feba97 feat(datafusion): Implement `insert_into` for
`IcebergTableProvider` (#1600)
new 960ecfc fix: await get catalog in datafusion integration test (#1635)
new ce7188c feat(writer): Make LocationGenerator partition-aware (#1625)
new e026b43 feat: bump datafusion to 48 (#1501)
new 4b9dc43 chore: Bump toolchain version to rust 1.87 and update
dependencies list (#1688)
new 87ce715 chore: update changelog and dependencies for 0.7.0 release
(#1713)
new 8f12434 refactor(writer): Refactor writers for the future
partitioning writers (#1657)
new 4d98e77 fix: ensure CoalescePartitionsExec is enabled for
IcebergCommitExec (#1723)
new ced4c0b Improve `IcebergCommitExec` to correctly populate
properties/schema (#1721)
new bea2fe2 feat(spec): add `table_properties.rs` to spec (#1733)
new 0c11819 feat(writer): Add clustered and fanout writer (#1735)
new 26e25c6 feat(datafusion): implement the project node to add the
partition columns (#1602)
new 07f0b67 refactor(arrow,datafusion): Reuse PartitionValueCalculator in
RecordBatchPartitionSplitter (#1781)
new ba6653c feat: Update Datafusion to v49 (#1704)
new 33bd90a chore: inline format args (#1805)
new 09ac6ae feat(datafusion): implement the partitioning node for
DataFusion to define the partitioning (#1620)
new 7cdec71 feat(datafusion): Add TaskWriter for DataFusion (#1769)
new 5bf2ff3 feat(datafusion): Support `INSERT INTO` partitioned tables
(#1827)
new 10b781d feat(datafusion): Split IcebergTableProvider into static and
non-static table provider (#1879)
new f86209e feat(datafusion): Add `sort_by_partition` to sort the input
partitioned data (#1618)
new 20f87db chore: bump MSRV to 1.88, fix warnings and clippy errors
(#1902)
new 043bf35 deps: bump DataFusion to 51, Arrow to 57, pyo to 0.26 (#1899)
new 50dfba6 Bump to version 0.8.0 (#1938)
new b90f91d feat: make FanoutWriter writer configurable (#1962)
new 5124214 feat(datafusion): Apply SortExec when writing in clustered
mode (#2005)
new a87d10f feat(datafusion): Add LIMIT pushdown support (#2006)
new 99ddb0a feat(spec): add table_properties() method to TableMetadata
(#2002)
new ab86392 feat(datafusion): Add schema validation for partition
projection (#2008)
new 851b18c feat(datafusion): Support `CREATE TABLE` for DataFusion
(#1972)
new 868ea8b feat(datafusion): Add Binary scalar value conversion for
predicate pushdown (#2048)
new 27a2259 feat(datafusion): Add support for DROP TABLE (#2033)
new 46a64c7 feat(datafusion): Add LIKE predicate pushdown for StartsWith
patterns (#2014)
new 0afd6d0 feat(datafusion): Add Boolean predicate pushdown support
(#2082)
new 5c2d83c feat(datafusion): Add Timestamp scalar value conversion for
predicate pushdown (#2069)
new 2f2afcf feat(datafusion): support isnan predicate pushdown to Iceberg
(#2142)
new 0b9ab78 feat!(storage): Integration Storage trait with FileIO (#2116)
new c35ca67 chore: Update MSRV to 1.91.0 (#2204)
new 3753ec0 chore: Bump version to 0.9.0, update dep list and changelog
(#2223)
new 33f4202 deps: upgrade DataFusion to 53.0, Arrow to 58 (#2206)
new f0b7875 Make `convert_filters_to_predicate` public (#2118)
new 09ead8c feat(datafusion): show pushed-down limit in IcebergTableScan
EXPLAIN output (#2360)
new a3077ed fix(tests): remove unnecessary `.to_string()` calls for
metadata_location (#2407)
new a972016 feat!(runtime): Support custom Runtime in Catalog (#2308)
new d9162fb feat: HuggingFace Hub storage backend and CDC table
properties (#2375)
new 29928a5 Use cargo publish to publish crates (#2489)
new 360f749 feat(ci): add cargo-public-api checks for public API
stability (#2525)
new 2baad09 chore(ci): omit blanket/auto-trait/auto-derived impls from
public API (#2542)
new c421c95 chore(refactor): add manifest list reader (#2541)
new ac36ed0 refactor!(writer): borrow TableMetadata in
DefaultLocationGenerator::new (#2604)
new 999cc09 chore: Bump MSRV to 1.94 (#2652)
new 46dfe94 feat(datafusion): push down isnan over NaN-preserving numeric
expressions (#2592)
new 45c3605 chore: Move dependency list generation from a release task to
a PR author responsibility (#2676)
new 05bed56 chore: Bump version to 0.10.0, update changelog (#2700)
new a030a1a chore: Update changelog and dependency lists for 0.10.0
(#2745)
new 21bd3b6 fix(datafusion): return single row with count 0 for empty
inserts (#2712)
new 7334077 fix(datafusion): reject non-append insert operations (#2714)
new e835b75 feat(writer): honor write.parquet.* via
ParquetWriterBuilder::from_table_properties (#2561)
new cdc31c1 chore: Update changelog for 0.10.0 (#2782)
new faba05f chore(deps): Bump DataFusion to 54.0.0 (#2648)
new 2c9644d feat: Variant Support (#2188)
new 823b746 chore(lint) check unused_qualifications (#2839)
new 49098eb feat: add history metadata table (#2825)
new 051b19b feat(parquet): wire in additional parquet writer settings
(#2887)
new b7819c1 fix (datafusion) IcebergWriteExec emits incorrect schema
(#2975)
new e140f9f feat(encryption) [15/N] write encrypted parquet data files
(#2701)
new fbf6a21 refactor: Add manifest reader (#2884)
new d1819cb refactor: Migrate TableProperties consumers to getters (#2989)
new 72b8fcb docs: fix references in public docs (#2937)
new 96783fc refactor: parse table properties lazily (#3030)
new 9b1a2e9 deps: upgrade to DataFusion 55.0 and Arrow/Parquet 59.2
(#2958)
The 121 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
