This is an automated email from the ASF dual-hosted git repository.
voonhous pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new 3eef646d3afd fix(hive): read the skeleton file when a bootstrap query
projects no columns (#19510)
3eef646d3afd is described below
commit 3eef646d3afdc692b84c432120900f86e7ac3a58
Author: Ranga Reddy <[email protected]>
AuthorDate: Thu Aug 27 19:10:34 2026 +0530
fix(hive): read the skeleton file when a bootstrap query projects no
columns (#19510)
A BootstrapBaseFileSplit carries two files: the split's own path is the
skeleton, inside the
table root, and getBootstrapFileSplit() is the external source file, which
is not.
createBootstrappingRecordReader opens one of them when only one is needed,
and it tested
"no Hudi meta column projected" first, which resolves to the external file.
SELECT COUNT(*) projects no columns at all, so both single-file conditions
hold and that
first branch won: Hive was handed a path outside the table root. Its
vectorized reader
resolves partition values by looking the split path up in
pathToPartitionInfo, which only
holds the table's own partition directories, so the query failed with
"cannot find dir = <external path> in pathToPartitionInfo".
Test the external-column condition first, so a projection of no columns
reads the skeleton.
Bootstrap keeps a one-to-one row correspondence between the two files, so
the count is
unchanged, and the skeleton is inside the table root and much smaller.
Scope, after review:
- This fixes the no-projection shape only. Projecting data columns still
opens the
external split, and so does the stitch branch, so a partitioned
bootstrap table still
fails for those - filed as #19643 with the
"set hive.vectorized.execution.enabled=false" workaround.
- It is not Hive-3 specific. hive-exec 2.3.10 ships the same vectorized
reader; what
differs is the default of hive.vectorized.execution.enabled, false in
hive-common 2.3.10
and true in 3.1.3 (javap on HiveConf$ConfVars: iconst_0 vs iconst_1).
Hive 2 with
vectorization enabled hits it too.
- MOR is reached only for a bootstrap file slice with no log files. When a
HoodieRealtimeBootstrapBaseFileSplit has delta logs,
addVirtualKeysProjection injects the
meta columns through the 3-arg addProjectionField, which does not
consult LIST_COLUMNS
and so always succeeds, leaving this branch unreachable.
The two booleans are now derived from the projected column names alone. The
previous code
zipped getReadColumnIDs with getReadColumnNames, which is the pairing
HoodieColumnProjectionUtils warns about in getReadColumnIDs itself: the id
list is
de-duplicated and parsed with Integer.parseInt while the name list is
neither, so a blank id
(the HIVE-22438 shape) threw NumberFormatException and a duplicated name
misaligned the zip.
The Integer half was never read.
Tests live in TestHoodieParquetInputFormat rather than a new class. The one
that matters goes
through getRecordReader: a bootstrap split whose skeleton holds 3 rows and
whose external file
holds 7, read with nothing projected, must yield 3. It yields 7 without the
fix. The remaining
three combinations are a parameterised truth table over the two booleans.
Closes #15676
---
.../hudi/hadoop/HoodieParquetInputFormat.java | 39 ++++++-
.../hudi/hadoop/TestHoodieParquetInputFormat.java | 127 +++++++++++++++++++++
2 files changed, 162 insertions(+), 4 deletions(-)
diff --git
a/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieParquetInputFormat.java
b/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieParquetInputFormat.java
index 6c10945fe39b..6a0a07976e8d 100644
---
a/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieParquetInputFormat.java
+++
b/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieParquetInputFormat.java
@@ -22,6 +22,7 @@ import org.apache.hudi.common.model.HoodieRecord;
import org.apache.hudi.common.schema.internal.InternalSchema;
import org.apache.hudi.common.util.HoodieStorageUtils;
import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
import org.apache.hudi.common.util.collection.Pair;
import org.apache.hudi.exception.HoodieException;
import org.apache.hudi.hadoop.avro.HoodieTimestampAwareParquetInputFormat;
@@ -210,10 +211,10 @@ public class HoodieParquetInputFormat extends
HoodieParquetInputFormatBase {
LOG.info("colNameWithTypes ={}, Num Entries ={}", colNameWithTypes,
colNameWithTypes.size());
- if (hoodieColsProjected.isEmpty()) {
- return getRecordReaderInternal(eSplit.getBootstrapFileSplit(), job,
reporter);
- } else if (externalColsProjected.isEmpty()) {
- return getRecordReaderInternal(split, job, reporter);
+ Option<FileSplit> singleSplit = resolveSingleFileSplit(eSplit,
!hoodieColsProjected.isEmpty(),
+ !externalColsProjected.isEmpty());
+ if (singleSplit.isPresent()) {
+ return getRecordReaderInternal(singleSplit.get(), job, reporter);
} else {
FileSplit rightSplit = eSplit.getBootstrapFileSplit();
// Hive PPD works at row-group level and only enabled when
hive.optimize.index.filter=true;
@@ -233,4 +234,34 @@ public class HoodieParquetInputFormat extends
HoodieParquetInputFormatBase {
true);
}
}
+
+ /**
+ * The single file backing this read, or empty when both files are needed
and have to be stitched.
+ *
+ * <p>A bootstrap split carries two paths: the split itself is the skeleton
file, which lives inside the
+ * table root, and {@code getBootstrapFileSplit()} is the external source
file, which does not.
+ *
+ * <p>The two "only one file is needed" cases both apply when a query
projects no columns at all, as
+ * {@code SELECT COUNT(*)} does, so the order they are tested in decides
which file is read. Prefer the
+ * skeleton: it is inside the table root, and bootstrap keeps a one-to-one
row correspondence with the
+ * external file, so a count over it is identical. Handing Hive a path
outside the table root breaks its
+ * vectorized reader, which derives partition values by looking the split
path up in
+ * {@code pathToPartitionInfo} (HUDI-5526).
+ *
+ * @param split the bootstrap split.
+ * @param anyHoodieColProjected whether the query projects any Hudi meta
column.
+ * @param anyExternalColProjected whether the query projects any column from
the external file.
+ */
+ @VisibleForTesting
+ static Option<FileSplit> resolveSingleFileSplit(BootstrapBaseFileSplit split,
+ boolean
anyHoodieColProjected,
+ boolean
anyExternalColProjected) {
+ if (!anyExternalColProjected) {
+ return Option.of(split);
+ } else if (!anyHoodieColProjected) {
+ return Option.of(split.getBootstrapFileSplit());
+ } else {
+ return Option.empty();
+ }
+ }
}
\ No newline at end of file
diff --git
a/hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/TestHoodieParquetInputFormat.java
b/hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/TestHoodieParquetInputFormat.java
index fe84fa8c8c56..a7782382835b 100644
---
a/hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/TestHoodieParquetInputFormat.java
+++
b/hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/TestHoodieParquetInputFormat.java
@@ -52,6 +52,7 @@ import org.apache.hadoop.io.ArrayWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.mapred.FileInputFormat;
+import org.apache.hadoop.mapred.FileSplit;
import org.apache.hadoop.mapred.InputSplit;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.RecordReader;
@@ -62,10 +63,14 @@ import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Disabled;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
+import java.nio.file.Files;
import java.nio.file.Paths;
import java.sql.Timestamp;
import java.text.SimpleDateFormat;
@@ -77,6 +82,7 @@ import java.util.ArrayList;
import java.util.Collections;
import java.util.Date;
import java.util.List;
+import java.util.stream.Stream;
import static
org.apache.hudi.common.testutils.HoodieTestUtils.COMMIT_METADATA_SER_DE;
import static
org.apache.hudi.common.testutils.HoodieTestUtils.INSTANT_FILE_NAME_GENERATOR;
@@ -86,6 +92,7 @@ import static
org.apache.hudi.common.testutils.SchemaTestUtil.getSchemaFromResou
import static
org.apache.hudi.hadoop.HoodieColumnProjectionUtils.READ_COLUMN_NAMES_CONF_STR;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertSame;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.junit.jupiter.api.Assertions.fail;
@@ -837,4 +844,124 @@ public class TestHoodieParquetInputFormat {
jobConf.set(HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key(), "true");
}
}
+
+ /**
+ * A bootstrap split carries two files: the split's own path is the
skeleton, inside the table root, and
+ * {@code getBootstrapFileSplit()} is the external source file, which is
not. A query projecting no columns
+ * at all - {@code SELECT COUNT(*)} - satisfies both "only one file is
needed" conditions at once, so the
+ * order they are tested in decides which file Hive is handed.
+ *
+ * <p>Handing Hive a path outside the table root breaks its vectorized
reader, which derives partition
+ * values by looking the split path up in {@code pathToPartitionInfo}
(HUDI-5526, #15676). Hive 2.3 ships
+ * the same reader but defaults {@code hive.vectorized.execution.enabled} to
false where Hive 3 defaults it
+ * to true, so this is gated by that config rather than by the Hive version.
+ *
+ * <p>Only the no-projection case is new behaviour: TestBootstrap and
TestOrcBootstrap drive the other
+ * three branches end to end, they have just been disabled (HUDI-7353) since
#10551.
+ */
+ @Test
+ public void testCountStarReadsSkeletonSoSplitPathStaysInsideTable() throws
IOException {
+ BootstrapBaseFileSplit split = bootstrapSplit();
+
+ Option<FileSplit> resolved =
HoodieParquetInputFormat.resolveSingleFileSplit(split, false, false);
+
+ assertTrue(resolved.isPresent(), "a query projecting no columns must
resolve to a single file");
+ assertSame(split, resolved.get(),
+ "it must be the skeleton, whose path is inside the table root");
+ }
+
+ /**
+ * The remaining three combinations, which behave the same before and after
the reorder: only meta columns
+ * needs the skeleton, only data columns needs the external file, and both
needs them stitched.
+ */
+ @ParameterizedTest
+ @MethodSource("singleFileSplitCases")
+ public void testSingleFileSplitSelection(boolean anyHoodieCol, boolean
anyExternalCol,
+ String expected) throws IOException
{
+ BootstrapBaseFileSplit split = bootstrapSplit();
+
+ Option<FileSplit> resolved =
+ HoodieParquetInputFormat.resolveSingleFileSplit(split, anyHoodieCol,
anyExternalCol);
+
+ if ("stitch".equals(expected)) {
+ assertFalse(resolved.isPresent(), "both files are needed, so the caller
must stitch them");
+ } else {
+ assertTrue(resolved.isPresent(), "a single file should have been
resolved");
+ assertSame("skeleton".equals(expected) ? split :
split.getBootstrapFileSplit(), resolved.get(),
+ "wrong file chosen for (anyHoodieCol=" + anyHoodieCol + ",
anyExternalCol=" + anyExternalCol + ")");
+ }
+ }
+
+ private static Stream<Arguments> singleFileSplitCases() {
+ return Stream.of(
+ Arguments.of(true, false, "skeleton"),
+ Arguments.of(false, true, "external"),
+ Arguments.of(true, true, "stitch"));
+ }
+
+ private static BootstrapBaseFileSplit bootstrapSplit() throws IOException {
+ return new BootstrapBaseFileSplit(
+ new FileSplit(new Path("/tbl/event_type=two/skeleton.parquet"), 0,
100, (String[]) null),
+ new FileSplit(new Path("/src/event_type=two/part-0.parquet"), 0, 100,
(String[]) null));
+ }
+
+ /**
+ * The end-to-end shape of HUDI-5526: a bootstrap split whose skeleton and
external file hold a different
+ * number of rows, read through {@link
HoodieParquetInputFormat#getRecordReader} with nothing projected.
+ * The reader must yield the skeleton's row count. On master it yields the
external file's.
+ *
+ * <p>No Hudi table is needed: {@code shouldUseFilegroupReader} excludes
{@code BootstrapBaseFileSplit}, so
+ * this falls straight through to {@code createBootstrappingRecordReader}.
+ */
+ @Test
+ public void testNoProjectionReaderReadsSkeletonRowCount() throws Exception {
+ HoodieSchema schema = SchemaTestUtil.getSchemaFromResource(getClass(),
"/test_timetype.avsc");
+ java.nio.file.Path skeletonFile = basePath.resolve("skeleton.parquet");
+ java.nio.file.Path externalFile = basePath.resolve("external.parquet");
+ int skeletonRows = 3;
+ int externalRows = 7;
+ writeParquet(skeletonFile, schema, skeletonRows);
+ writeParquet(externalFile, schema, externalRows);
+
+ jobConf.set(HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key(), "false");
+ jobConf.set(IOConstants.COLUMNS,
"test_timestamp,test_long,test_date,_hoodie_commit_time,_hoodie_commit_seqno");
+ jobConf.set(IOConstants.COLUMNS_TYPES,
"timestamp,bigint,date,string,string");
+ // SELECT COUNT(*): Hive projects no columns at all.
+ jobConf.set(READ_COLUMN_NAMES_CONF_STR, "");
+ jobConf.set(HoodieColumnProjectionUtils.READ_COLUMN_IDS_CONF_STR, "");
+
+ BootstrapBaseFileSplit split = new BootstrapBaseFileSplit(
+ new FileSplit(new Path(skeletonFile.toString()), 0,
Files.size(skeletonFile), (String[]) null),
+ new FileSplit(new Path(externalFile.toString()), 0,
Files.size(externalFile), (String[]) null));
+
+ RecordReader<NullWritable, ArrayWritable> reader =
inputFormat.getRecordReader(split, jobConf, null);
+ try {
+ NullWritable key = reader.createKey();
+ ArrayWritable value = reader.createValue();
+ int rows = 0;
+ while (reader.next(key, value)) {
+ rows++;
+ }
+ assertEquals(skeletonRows, rows,
+ "a no-projection read must come from the skeleton file, whose path
is inside the table root");
+ } finally {
+ reader.close();
+ }
+ }
+
+ private static void writeParquet(java.nio.file.Path file, HoodieSchema
schema, int numRows) throws IOException {
+ try (AvroParquetWriter parquetWriter =
+ new AvroParquetWriter(new Path(file.toString()),
schema.toAvroSchema())) {
+ for (int i = 0; i < numRows; i++) {
+ GenericData.Record record = new
GenericData.Record(schema.toAvroSchema());
+ record.put("test_timestamp", (long) i);
+ record.put("test_long", (long) i);
+ record.put("test_date", i);
+ record.put("_hoodie_commit_time", "20160628071126");
+ record.put("_hoodie_commit_seqno", "20160628071126_" + i);
+ parquetWriter.write(record);
+ }
+ }
+ }
+
}
