rangareddy commented on issue #15676:
URL: https://github.com/apache/hudi/issues/15676#issuecomment-5178522324
I traced this on current master. The root cause is readable from the code
and matches the reported paths exactly, so recording it here — including a
likely workaround you can try today.
### Root cause
`BootstrapBaseFileSplit` carries **two** paths, and the constructor makes
the distinction easy to miss:
```java
public BootstrapBaseFileSplit(FileSplit baseSplit, FileSplit
bootstrapFileSplit) {
super(baseSplit.getPath(), baseSplit.getStart(), baseSplit.getLength(),
baseSplit.getLocations());
this.bootstrapFileSplit = bootstrapFileSplit;
}
```
So **the split itself is the Hudi skeleton file**, inside the table root,
while **`getBootstrapFileSplit()` is the external source file**, outside it.
`HoodieParquetInputFormat.createBootstrappingRecordReader` then picks
between them by projection:
```java
if (hoodieColsProjected.isEmpty()) {
return getRecordReaderInternal(eSplit.getBootstrapFileSplit(), job,
reporter); // external, OUTSIDE table root
} else if (externalColsProjected.isEmpty()) {
return getRecordReaderInternal(split, job, reporter);
// skeleton, inside table root
} else {
// stitch both
}
```
For `SELECT COUNT(*)` no columns are projected at all, so **both** lists are
empty and the first branch wins. Hudi therefore hands Hive a split whose path
lies outside the table.
On Hive 3 that path is then vectorized:
```
VectorizedParquetRecordReader.<init> -> initPartitionValues
-> VectorizedRowBatchCtx.getPartitionValues
-> HiveFileFormatUtils.getFromPathRecursively
```
`getFromPathRecursively` derives the partition values by looking the split
path up in `pathToPartitionInfo`, which only contains the **Hudi** table's
partition directories. The external path is not in it, which is precisely your
error:
```
cannot find dir =
[s3://my-bucket/test-data/hudi/parquet-source-tables/hive_style_partitioned_tb/event_type=two/part-00000-...parquet]
in pathToPartitionInfo:
[[s3://my-bucket/hudi-table/test_bootstrap_hive_partitionedrt/event_type=one],
[s3://my-bucket/hudi-table/test_bootstrap_hive_partitionedrt/event_type=two]]
```
That also explains "the same query works with Hive2": Hive 2 did not take
the vectorized parquet path, so `initPartitionValues` was never called and the
out-of-table path was never looked up.
### Likely workaround
```sql
set hive.vectorized.execution.enabled=false;
SELECT COUNT(*) FROM HUDI_BOOTSTRAP_TABLE;
```
That avoids `VectorizedParquetRecordReader` entirely, so the partition-value
lookup that fails never runs. I have not been able to confirm it on Hive 3 +
Tez myself, so please treat it as a strong hypothesis rather than a verified
fix — but it follows directly from the stack trace, and it costs nothing to try.
### This is not actually specific to COUNT
Worth correcting the title's implication: the branch condition is "no Hudi
meta columns projected", not "count". So a plain projection of data columns —
`SELECT <some_data_col> FROM HUDI_BOOTSTRAP_TABLE` — should take the same
branch and fail the same way on Hive 3. If you have a moment, confirming that
would be useful, because it changes the shape of the fix.
### On fixing it
Only the `COUNT(*)` case has a cheap fix. When **both** projection lists are
empty nothing needs the external file, and bootstrap guarantees a 1:1 row
correspondence between skeleton and external file — so reading the skeleton
would give the same count from a path that *is* inside the table root, and
Hive's lookup would succeed. That is a precedence change in the `if`/`else if`
above.
Queries that genuinely project data columns cannot be fixed that way, since
the data only exists in the external file. Those need either vectorization
disabled for bootstrap splits, or the partition values supplied rather than
derived from the path.
Two further notes for whoever picks this up:
- This legacy path is **still live on 1.x**.
`HoodieInputFormatUtils.shouldUseFilegroupReader` ends with `&& !(split
instanceof BootstrapBaseFileSplit)`, so bootstrap tables never reach the new
file-group reader and always land in `createBootstrappingRecordReader`.
- It has **no test coverage at all**: nothing in the test tree references
`BootstrapBaseFileSplit`, `createBootstrappingRecordReader` or
`BootstrapColumnStichingRecordReader`. The branch selection above is
unit-testable without Hive, so that would be the place to start before changing
the precedence.
I have not raised a PR, because the end-to-end behaviour needs a Hive 3 +
Tez bootstrap table to verify and I do not have one. Happy to do the precedence
change plus first unit coverage of the branch selection if a maintainer thinks
that is the right direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
