This is an automated email from the ASF dual-hosted git repository.
dependabot[bot] pushed a change to branch
dependabot/github_actions/codecov/codecov-action-7
in repository https://gitbox.apache.org/repos/asf/hudi-rs.git
discard 80c3e585 build(deps): bump codecov/codecov-action from 6 to 7
add 5ad7c4a1 ci: use action refs the ASF allowlist accepts (#674)
add f32b0eee fix: stop .gitignore from swallowing test fixtures (#676)
add 9b09398d feat(core): select the file group reader by version,
defaulting to 2 (#677)
add 75b188ef feat(core): serve merge-on-read reads with file group reader
version 2 (#660)
add c7e3f033 test(core): cover event-time ordering losing in the gold
corpus (#680)
add 3fd4e88d fix(core): gate the log-block scan on instant state (#678)
add b77f5ce5 fix(core): fold partial updates in both ordering directions
(#679)
add 53a403c8 chore: let consumers opt out of the merge map's on-disk tier
(#681)
add e8ff14a1 perf(core): copy an HFile record rather than its whole block
per record (#697)
add 3f41255b perf(core): bound the merge chunk rather than moving the
merge off the executor (#689)
add 874d9430 build(deps): bump actions/cache from 5 to 6 (#635)
add fa43124e build(deps): bump actions/checkout from 6 to 7 (#632)
add 5e8db0ee build(deps-dev): bump pytest from 9.0.3 to 9.1.1 in /python
(#634)
add 090afe4e build(deps): bump codecov/codecov-action from 6 to 7
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (80c3e585)
\
N -- N -- N
refs/heads/dependabot/github_actions/codecov/codecov-action-7 (090afe4e)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../workflows/{pr.yml => asf-allowlist-check.yml} | 30 +-
.github/workflows/ci.yml | 94 +-
.github/workflows/code.yml | 4 +-
.github/workflows/pr.yml | 2 +-
.github/workflows/release.yml | 10 +-
.gitignore | 8 +
AGENTS.md | 52 +-
Cargo.toml | 13 +
crates/core/Cargo.toml | 26 +
crates/core/schemas/HoodieDeleteRecord.avsc | 24 +-
crates/core/schemas/HoodieDeleteRecordList.avsc | 129 +-
.../core/src/avro_to_arrow/arrow_array_reader.rs | 932 ---
crates/core/src/avro_to_arrow/mod.rs | 1 -
crates/core/src/avro_to_arrow/schema.rs | 111 +-
crates/core/src/config/internal.rs | 25 +
crates/core/src/config/mod.rs | 9 +
crates/core/src/config/read.rs | 182 +-
crates/core/src/config/table.rs | 14 +-
crates/core/src/error.rs | 4 +
crates/core/src/file_group/base_file/mod.rs | 14 +
crates/core/src/file_group/base_file/parquet.rs | 318 +-
crates/core/src/file_group/base_file/reader.rs | 52 +-
crates/core/src/file_group/builder.rs | 319 +-
crates/core/src/file_group/file_slice.rs | 150 +-
crates/core/src/file_group/log_file/avro.rs | 279 +-
crates/core/src/file_group/log_file/content.rs | 919 ++-
crates/core/src/file_group/log_file/log_block.rs | 124 +
.../core/src/file_group/log_file/memory_bench.rs | 202 +
crates/core/src/file_group/log_file/mod.rs | 20 +-
crates/core/src/file_group/log_file/reader.rs | 1045 ++-
crates/core/src/file_group/log_file/scanner.rs | 2 +-
crates/core/src/file_group/mod.rs | 228 +-
crates/core/src/file_group/reader.rs | 1361 +++-
crates/core/src/file_group/reader_v2/adapter.rs | 294 +
.../src/file_group/reader_v2/buffer/key_based.rs | 8069 ++++++++++++++++++++
.../core/src/file_group/reader_v2/buffer/loader.rs | 517 ++
crates/core/src/file_group/reader_v2/buffer/mod.rs | 314 +
.../file_group/reader_v2/buffer/position_based.rs | 1511 ++++
.../file_group/reader_v2/buffer/record_buffer.rs | 544 ++
.../reader_v2/buffer/record_positions.rs | 225 +
.../file_group/reader_v2/buffer/row_extraction.rs | 722 ++
.../file_group/reader_v2/buffer/spillable_map.rs | 2619 +++++++
.../src/file_group/reader_v2/buffered_record.rs | 637 ++
.../reader_v2/buffered_record_converter.rs | 45 +
.../src/file_group/reader_v2/delete_context.rs | 342 +
crates/core/src/file_group/reader_v2/engine.rs | 2220 ++++++
crates/core/src/file_group/reader_v2/gaps.rs | 252 +
crates/core/src/file_group/reader_v2/harness.rs | 1020 +++
.../core/src/file_group/reader_v2/harness_tests.rs | 2270 ++++++
.../core/src/file_group/reader_v2/input_split.rs | 247 +
.../core/src/file_group/reader_v2/iterator_mode.rs | 97 +
.../src/file_group/reader_v2/log_record_reader.rs | 2283 ++++++
.../src/file_group/reader_v2/memory_limit_tests.rs | 296 +
.../src/file_group/reader_v2/merge_iterator.rs | 1485 ++++
.../reader_v2/merged_log_record_reader.rs | 697 ++
crates/core/src/file_group/reader_v2/mod.rs | 75 +
.../src/file_group/reader_v2/output_converter.rs | 449 ++
crates/core/src/file_group/reader_v2/profiling.rs | 46 +
crates/core/src/file_group/reader_v2/read_stats.rs | 82 +
.../src/file_group/reader_v2/reader_context.rs | 454 ++
.../src/file_group/reader_v2/reader_parameters.rs | 42 +
.../src/file_group/reader_v2/record_context.rs | 2756 +++++++
.../core/src/file_group/reader_v2/record_merger.rs | 820 ++
crates/core/src/file_group/reader_v2/resolver.rs | 966 +++
crates/core/src/file_group/reader_v2/row_serde.rs | 254 +
.../src/file_group/reader_v2/schema_handler.rs | 1367 ++++
.../src/file_group/reader_v2/update_processor.rs | 193 +
crates/core/src/file_group/record_batches.rs | 30 +-
crates/core/src/hfile/key.rs | 121 +-
crates/core/src/merge/record_merger.rs | 58 +-
crates/core/src/metadata/table/mod.rs | 22 +-
crates/core/src/metadata/table/reader.rs | 282 +
crates/core/src/schema/avro_schema_utils.rs | 493 ++
crates/core/src/schema/batch_evolution.rs | 1259 +++
crates/core/src/schema/delete.rs | 215 +-
crates/core/src/schema/extended_promotion.rs | 665 ++
crates/core/src/schema/mod.rs | 4 +
crates/core/src/schema/parquet_list_norm.rs | 319 +
crates/core/src/schema/resolver.rs | 16 +
crates/core/src/storage/mod.rs | 34 +-
crates/core/src/storage/reader.rs | 497 +-
crates/core/src/table/builder.rs | 98 +-
crates/core/src/table/fs_view.rs | 52 +-
crates/core/src/table/listing.rs | 37 +-
crates/core/src/table/mod.rs | 1174 ++-
crates/core/src/timeline/completion_time.rs | 18 +-
crates/core/src/timeline/instant.rs | 85 +-
crates/core/src/timeline/loader.rs | 80 +
crates/core/src/timeline/mod.rs | 192 +-
crates/core/src/timeline/selector.rs | 254 +-
crates/core/src/timeline/view.rs | 186 +-
crates/core/src/util/arrow.rs | 8 +-
.../data/i3/legacy_2level_repeated_map.parquet | Bin 0 -> 13270 bytes
.../core/tests/data/i3/modern_3level_map.parquet | Bin 0 -> 10608 bytes
crates/core/tests/gold_parity_tests.rs | 1019 +++
crates/core/tests/reader_concurrency_tests.rs | 347 +
crates/core/tests/table_read_tests.rs | 508 +-
crates/datafusion/Cargo.toml | 12 +-
crates/datafusion/src/hudi_exec.rs | 21 +-
crates/datafusion/src/lib.rs | 16 +-
crates/hudi/Cargo.toml | 15 +-
crates/test/Cargo.toml | 9 +
crates/test/README.md | 14 +
.../mor/avro/gold_options.scala | 280 +
.../mor/avro/table_all_data_types.zip | Bin 0 -> 47647 bytes
.../mor/avro/table_column_projection.zip | Bin 0 -> 43480 bytes
.../mor/avro/table_corrupt_tail_block.zip | Bin 0 -> 27775 bytes
.../mor/avro/table_delete_ord_decimal.zip | Bin 0 -> 24327 bytes
.../mor/avro/table_delete_ord_double.zip | Bin 0 -> 24010 bytes
.../mor/avro/table_delete_ord_int.zip | Bin 0 -> 23482 bytes
.../mor/avro/table_delete_ord_long.zip | Bin 0 -> 23890 bytes
.../mor/avro/table_delete_ord_string.zip | Bin 0 -> 23739 bytes
.../mor/avro/table_delete_ord_timestamp.zip | Bin 0 -> 24480 bytes
.../mor/avro/table_duplicate_keys.scala | 111 +
.../mor/avro/table_duplicate_keys.zip | Bin 0 -> 17609 bytes
.../mor/avro/table_event_time_stale.sql | 69 +
.../mor/avro/table_event_time_stale.zip | Bin 0 -> 37278 bytes
.../mor/avro/table_evo_add_col.zip | Bin 0 -> 125693 bytes
.../mor/avro/table_evo_promotion.zip | Bin 0 -> 114020 bytes
.../mor/avro/table_hfile_log_block.zip | Bin 0 -> 10865 bytes
.../mor/avro/table_log_compaction.zip | Bin 0 -> 30229 bytes
.../mor/avro/table_log_only.zip | Bin 0 -> 23937 bytes
.../mor/avro/table_null_containers.zip | Bin 0 -> 38514 bytes
.../mor/avro/table_parquet_log_block.zip | Bin 0 -> 32213 bytes
.../mor/avro/table_partial_update.zip | Bin 0 -> 24790 bytes
.../mor/avro/table_partial_update_event_time.sql | 74 +
.../mor/avro/table_partial_update_event_time.zip | Bin 0 -> 16147 bytes
.../mor/avro/table_uncommitted_log.sql | 78 +
.../mor/avro/table_uncommitted_log_v6.zip | Bin 0 -> 24847 bytes
.../mor/avro/table_uncommitted_log_v9.zip | Bin 0 -> 30414 bytes
.../mor/avro/v6_trips_8i1u.zip | Bin 28861 -> 43668 bytes
.../mor/avro/v6_trips_8i3d.zip | Bin 28516 -> 42817 bytes
.../mor/avro/v8_mor_boundary_windows.gold.scala | 52 +
.../mor/avro/v8_mor_boundary_windows.sql | 48 +
.../mor/avro/v8_mor_boundary_windows.zip | Bin 0 -> 37974 bytes
.../mor/avro/v8_trips_8i3u1d.zip | Bin 468520 -> 458184
bytes
.../mor/avro/v9_mor_8i4u_commit_time.zip | Bin 0 -> 38584 bytes
.../avro/v9_mor_compacted_incremental.gold.scala | 70 +
.../mor/avro/v9_mor_compacted_incremental.sql | 72 +
.../mor/avro/v9_mor_compacted_incremental.zip | Bin 0 -> 43318 bytes
.../mor/avro/v9_mor_nonpart_3commits.zip | Bin 0 -> 28475 bytes
.../cow/v6_complexkeygen_hivestyle.zip | Bin 42914 -> 41900 bytes
crates/test/data/sample_table/cow/v6_empty.zip | Bin 2258 -> 2371 bytes
.../data/sample_table/cow/v6_nonpartitioned.zip | Bin 24851 -> 25426 bytes
.../v6_simplekeygen_hivestyle_no_metafields.zip | Bin 17950 -> 19260 bytes
.../cow/v6_simplekeygen_nonhivestyle.zip | Bin 38375 -> 37976 bytes
...v6_simplekeygen_nonhivestyle_overwritetable.zip | Bin 36037 -> 37460 bytes
.../cow/v6_timebasedkeygen_nonhivestyle.zip | Bin 49127 -> 47642 bytes
.../cow/v8_complexkeygen_hivestyle.zip | Bin 43000 -> 42629 bytes
crates/test/data/sample_table/cow/v8_empty.zip | Bin 6066 -> 5277 bytes
.../data/sample_table/cow/v8_nonpartitioned.zip | Bin 26699 -> 27451 bytes
.../v8_simplekeygen_hivestyle_no_metafields.zip | Bin 28076 -> 27855 bytes
.../cow/v8_simplekeygen_nonhivestyle.zip | Bin 40168 -> 40070 bytes
.../cow/v9_timebasedkeygen_epochmillis.zip | Bin 201769 -> 198528
bytes
.../cow/v9_timebasedkeygen_nonhivestyle.zip | Bin 192801 -> 189598
bytes
.../cow/v9_timebasedkeygen_unixtimestamp.zip | Bin 206530 -> 203649
bytes
.../data/sample_table/cow/v9_txns_complex_meta.zip | Bin 711746 -> 699782
bytes
.../sample_table/cow/v9_txns_complex_nometa.zip | Bin 106586 -> 106426
bytes
.../data/sample_table/cow/v9_txns_nonpart_meta.zip | Bin 564571 -> 554519
bytes
.../sample_table/cow/v9_txns_nonpart_nometa.zip | Bin 81181 -> 81444 bytes
.../data/sample_table/cow/v9_txns_simple_meta.zip | Bin 712093 -> 699930
bytes
.../sample_table/cow/v9_txns_simple_nometa.zip | Bin 104308 -> 104005
bytes
.../sample_table/cow/v9_txns_simple_overwrite.zip | Bin 215146 -> 211956
bytes
.../mor/avro/v9_nonpartitioned_rollback.zip | Bin 100397 -> 98943 bytes
.../mor/avro/v9_timebasedkeygen_nonhivestyle.zip | Bin 242527 -> 238618
bytes
.../sample_table/mor/avro/v9_txns_complex_meta.zip | Bin 731013 -> 718435
bytes
.../mor/avro/v9_txns_complex_nometa.zip | Bin 100713 -> 100299
bytes
.../sample_table/mor/avro/v9_txns_nonpart_meta.zip | Bin 570239 -> 559644
bytes
.../mor/avro/v9_txns_nonpart_nometa.zip | Bin 68385 -> 68432 bytes
.../sample_table/mor/avro/v9_txns_simple_meta.zip | Bin 733891 -> 721073
bytes
.../mor/avro/v9_txns_simple_nometa.zip | Bin 98645 -> 98103 bytes
.../mor/avro/v9_txns_simple_overwrite.zip | Bin 215161 -> 211978
bytes
.../mor/parquet/v6_complexkeygen_hivestyle.zip | Bin 42137 -> 41107 bytes
.../data/sample_table/mor/parquet/v6_empty.zip | Bin 5885 -> 5062 bytes
.../sample_table/mor/parquet/v6_nonpartitioned.zip | Bin 28459 -> 28935 bytes
.../mor/parquet/v6_nonpartitioned_rollback.zip | Bin 32598 -> 32214 bytes
.../v6_simplekeygen_hivestyle_no_metafields.zip | Bin 28077 -> 27470 bytes
.../mor/parquet/v6_simplekeygen_nonhivestyle.zip | Bin 37571 -> 37180 bytes
...v6_simplekeygen_nonhivestyle_overwritetable.zip | Bin 48409 -> 46787 bytes
crates/test/src/gold.rs | 575 ++
crates/test/src/gold_options.rs | 373 +
crates/test/src/lib.rs | 366 +-
docs/reader-spec.md | 4 +-
python/pyproject.toml | 2 +-
python/src/internal.rs | 27 +-
python/tests/test_table_incremental_read.py | 50 +-
186 files changed, 49384 insertions(+), 1940 deletions(-)
copy .github/workflows/{pr.yml => asf-allowlist-check.yml} (63%)
delete mode 100644 crates/core/src/avro_to_arrow/arrow_array_reader.rs
create mode 100644 crates/core/src/file_group/log_file/memory_bench.rs
create mode 100644 crates/core/src/file_group/reader_v2/adapter.rs
create mode 100644 crates/core/src/file_group/reader_v2/buffer/key_based.rs
create mode 100644 crates/core/src/file_group/reader_v2/buffer/loader.rs
create mode 100644 crates/core/src/file_group/reader_v2/buffer/mod.rs
create mode 100644
crates/core/src/file_group/reader_v2/buffer/position_based.rs
create mode 100644 crates/core/src/file_group/reader_v2/buffer/record_buffer.rs
create mode 100644
crates/core/src/file_group/reader_v2/buffer/record_positions.rs
create mode 100644
crates/core/src/file_group/reader_v2/buffer/row_extraction.rs
create mode 100644 crates/core/src/file_group/reader_v2/buffer/spillable_map.rs
create mode 100644 crates/core/src/file_group/reader_v2/buffered_record.rs
create mode 100644
crates/core/src/file_group/reader_v2/buffered_record_converter.rs
create mode 100644 crates/core/src/file_group/reader_v2/delete_context.rs
create mode 100644 crates/core/src/file_group/reader_v2/engine.rs
create mode 100644 crates/core/src/file_group/reader_v2/gaps.rs
create mode 100644 crates/core/src/file_group/reader_v2/harness.rs
create mode 100644 crates/core/src/file_group/reader_v2/harness_tests.rs
create mode 100644 crates/core/src/file_group/reader_v2/input_split.rs
create mode 100644 crates/core/src/file_group/reader_v2/iterator_mode.rs
create mode 100644 crates/core/src/file_group/reader_v2/log_record_reader.rs
create mode 100644 crates/core/src/file_group/reader_v2/memory_limit_tests.rs
create mode 100644 crates/core/src/file_group/reader_v2/merge_iterator.rs
create mode 100644
crates/core/src/file_group/reader_v2/merged_log_record_reader.rs
create mode 100644 crates/core/src/file_group/reader_v2/mod.rs
create mode 100644 crates/core/src/file_group/reader_v2/output_converter.rs
create mode 100644 crates/core/src/file_group/reader_v2/profiling.rs
create mode 100644 crates/core/src/file_group/reader_v2/read_stats.rs
create mode 100644 crates/core/src/file_group/reader_v2/reader_context.rs
create mode 100644 crates/core/src/file_group/reader_v2/reader_parameters.rs
create mode 100644 crates/core/src/file_group/reader_v2/record_context.rs
create mode 100644 crates/core/src/file_group/reader_v2/record_merger.rs
create mode 100644 crates/core/src/file_group/reader_v2/resolver.rs
create mode 100644 crates/core/src/file_group/reader_v2/row_serde.rs
create mode 100644 crates/core/src/file_group/reader_v2/schema_handler.rs
create mode 100644 crates/core/src/file_group/reader_v2/update_processor.rs
create mode 100644 crates/core/src/metadata/table/reader.rs
create mode 100644 crates/core/src/schema/avro_schema_utils.rs
create mode 100644 crates/core/src/schema/batch_evolution.rs
create mode 100644 crates/core/src/schema/extended_promotion.rs
create mode 100644 crates/core/src/schema/parquet_list_norm.rs
create mode 100644 crates/core/tests/data/i3/legacy_2level_repeated_map.parquet
create mode 100644 crates/core/tests/data/i3/modern_3level_map.parquet
create mode 100644 crates/core/tests/gold_parity_tests.rs
create mode 100644 crates/core/tests/reader_concurrency_tests.rs
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/gold_options.scala
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/table_all_data_types.zip
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/table_column_projection.zip
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/table_corrupt_tail_block.zip
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/table_delete_ord_decimal.zip
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/table_delete_ord_double.zip
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/table_delete_ord_int.zip
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/table_delete_ord_long.zip
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/table_delete_ord_string.zip
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/table_delete_ord_timestamp.zip
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/table_duplicate_keys.scala
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/table_duplicate_keys.zip
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/table_event_time_stale.sql
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/table_event_time_stale.zip
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/table_evo_add_col.zip
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/table_evo_promotion.zip
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/table_hfile_log_block.zip
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/table_log_compaction.zip
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/table_log_only.zip
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/table_null_containers.zip
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/table_parquet_log_block.zip
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/table_partial_update.zip
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/table_partial_update_event_time.sql
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/table_partial_update_event_time.zip
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/table_uncommitted_log.sql
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/table_uncommitted_log_v6.zip
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/table_uncommitted_log_v9.zip
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/v8_mor_boundary_windows.gold.scala
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/v8_mor_boundary_windows.sql
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/v8_mor_boundary_windows.zip
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/v9_mor_8i4u_commit_time.zip
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/v9_mor_compacted_incremental.gold.scala
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/v9_mor_compacted_incremental.sql
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/v9_mor_compacted_incremental.zip
create mode 100644
crates/test/data/quickstart_trips_table/mor/avro/v9_mor_nonpart_3commits.zip
create mode 100644 crates/test/src/gold.rs
create mode 100644 crates/test/src/gold_options.rs
