Re: [PR] fix: handle non-partitioned table's special path value in MDT files partition [hudi-rs]
xushiyan merged PR #505: URL: https://github.com/apache/hudi-rs/pull/505 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix: handle non-partitioned table's special path value in MDT files partition [hudi-rs]
codecov[bot] commented on PR #505: URL: https://github.com/apache/hudi-rs/pull/505#issuecomment-3704627945 ## [Codecov](https://app.codecov.io/gh/apache/hudi-rs/pull/505?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report :x: Patch coverage is `92.5%` with `3 lines` in your changes missing coverage. Please review. :white_check_mark: Project coverage is 87.68%. Comparing base ([`3e3a2cf`](https://app.codecov.io/gh/apache/hudi-rs/commit/3e3a2cf11bb72b8d12d1271d56f82f7385304d78?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`2264d40`](https://app.codecov.io/gh/apache/hudi-rs/commit/2264d401d48c5dc270551f5c30ee28ca66fa3679?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)). :warning: Report is 5 commits behind head on main. | [Files with missing lines](https://app.codecov.io/gh/apache/hudi-rs/pull/505?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines | |---|---|---| | [crates/test/src/lib.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/505?src=pr&el=tree&filepath=crates%2Ftest%2Fsrc%2Flib.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL3Rlc3Qvc3JjL2xpYi5ycw==) | 87.50% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/505?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [crates/core/src/file\_group/reader.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/505?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Ffile_group%2Freader.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL2ZpbGVfZ3JvdXAvcmVhZGVyLnJz) | 90.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/505?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Additional details and impacted files ```diff @@Coverage Diff @@ ## main #505 +/- ## == + Coverage 87.59% 87.68% +0.08% == Files 65 65 Lines4104 4142 +38 == + Hits 3595 3632 +37 - Misses509 510 +1 ``` [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/hudi-rs/pull/505?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). :rocket: New features to boost your workflow: - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix: handle non-partitioned table's special path value in MDT files partition [hudi-rs]
Copilot commented on code in PR #505:
URL: https://github.com/apache/hudi-rs/pull/505#discussion_r2656955186
##
crates/core/tests/table_read_tests.rs:
##
@@ -550,3 +550,140 @@ mod v8_tables {
}
}
}
+
+/// Test module for tables with metadata table (MDT) enabled.
+/// These tests verify MDT-accelerated file listing and partition
normalization.
+mod mdt_enabled_tables {
+use super::*;
+use hudi_core::table::partition::PartitionPruner;
+
+mod snapshot_queries {
+use super::*;
+
+/// Test reading a V8 MOR non-partitioned table with MDT enabled.
+/// Verifies:
+/// 1. Table can be read correctly via MDT file listing
+/// 2. MDT partition key normalization ("." -> "") works correctly
+/// 3. File slices are retrieved correctly from MDT
+#[test]
+fn test_v8_nonpartitioned_with_mdt() -> Result<()> {
+let base_url = SampleTableMdt::V8Nonpartitioned.url_to_mor_avro();
+let hudi_table = Table::new_blocking(base_url.path())?;
+
+// Verify MDT is enabled
+assert!(
+hudi_table.is_metadata_table_enabled(),
+"Metadata table should be enabled"
+);
+
+// Get file slices - this uses MDT file listing
+let file_slices =
hudi_table.get_file_slices_blocking(empty_filters())?;
+
+// Should have file slices for the non-partitioned table
+assert!(
+!file_slices.is_empty(),
+"Should have file slices from MDT listing"
+);
+
+// All file slices should be in the root partition (empty string)
+for fs in &file_slices {
+assert_eq!(
+&fs.partition_path, "",
+"Non-partitioned table should have files in root partition"
+);
+}
+
+Ok(())
+}
+
+/// Test MDT partition key normalization for non-partitioned tables.
+/// The metadata table stores "." as partition key, but external API
should see "".
+/// For non-partitioned tables, we use a fast path that directly
fetches "." without
+/// going through __all_partitions__ lookup.
+#[test]
+fn test_v8_nonpartitioned_mdt_partition_normalization() -> Result<()> {
+let base_url = SampleTableMdt::V8Nonpartitioned.url_to_mor_avro();
+let hudi_table = Table::new_blocking(base_url.path())?;
+
+// Read MDT files partition records
+let partition_pruner = PartitionPruner::empty();
+let records =
+
hudi_table.read_metadata_table_files_partition_blocking(&partition_pruner)?;
+
+// For non-partitioned tables, the fast path only fetches the
files record.
+// __all_partitions__ is not fetched to avoid redundant HFile
lookup.
+assert_eq!(
+records.len(),
+1,
+"Non-partitioned table fast path should only fetch files
record"
+);
+
+// The files record should be keyed by "" (empty string)
+// not "." (which is the internal MDT representation)
+assert!(
+records.contains_key(""),
+"Non-partitioned table should have files record with empty
string key"
+);
+assert!(
+!records.contains_key("."),
+"Non-partitioned table should NOT have files record with '.'
key after normalization"
+);
+
+// Verify the files record has actual file entries
+let files_record = records.get("").unwrap();
+assert!(
+!files_record.files.is_empty(),
+"Files record should contain file entries"
+);
+
+Ok(())
+}
+
+/// Test pruned MDT lookup for non-partitioned tables.
+/// Verifies that the fast path directly fetches "." for
non-partitioned tables,
+/// skipping the __all_partitions__ lookup entirely.
+#[test]
+fn test_v8_nonpartitioned_mdt_pruned_lookup() -> Result<()> {
+use hudi_core::expr::filter::from_str_tuples;
+
+let base_url = SampleTableMdt::V8Nonpartitioned.url_to_mor_avro();
+let hudi_table = Table::new_blocking(base_url.path())?;
+let partition_schema = hudi_table.get_partition_schema_blocking()?;
+
+// Create a partition pruner that would match the non-partitioned
table's root partition.
+// For non-partitioned tables, the partition path is "" (empty
string).
+let filters = from_str_tuples([("", "=", "")])?;
+let partition_pruner = PartitionPruner::new(
+&filters,
+&partition_schema,
+hudi_table.hudi_configs.as_ref(),
+)?;
+
+// For non-partitioned tables,
[PR] fix: handle non-partitioned table's special path value in MDT files partition [hudi-rs]
xushiyan opened a new pull request, #505: URL: https://github.com/apache/hudi-rs/pull/505 ## Description MDT files stores `.` as the partition path key for non-partitioned table. Fix the handling of this special case. ## How are the changes test-covered - [ ] N/A - [ ] Automated tests (unit and/or integration tests) - [ ] Manual tests - [ ] Details are described below -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
