ad1happy2go commented on code in PR #19005:
URL: https://github.com/apache/hudi/pull/19005#discussion_r4571642104
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala:
##########
@@ -353,6 +388,48 @@ class HoodieFileGroupReaderBasedFileFormat(tablePath:
String,
}
}
+ /**
+ * The incremental window end (latest commit time in the window), extracted
from the `In` filter on
+ * the commit-time metadata field in `requiredFilters`. Used to bound the
merge inputs so that log
+ * blocks committed after the window are not merged. Returns None when no
such filter is present.
+ */
+ private def incrementalWindowEnd(requiredFilters: Seq[Filter]):
Option[String] = {
+ val commitTimes: Seq[String] = requiredFilters.collect {
+ case In(attr, values) if attr == HoodieRecord.COMMIT_TIME_METADATA_FIELD
=>
+ values.filter(_ != null).map(_.toString).toSeq
+ }.flatten
+ if (commitTimes.isEmpty) None else Some(commitTimes.max)
+ }
+
+ /**
+ * Builds a predicate over merged rows that keeps only the records whose
`_hoodie_commit_time`
+ * falls within the incremental query window encoded by `requiredFilters`
(an `In` filter on the
+ * commit-time metadata field, produced by the incremental relations).
Returns None when no such
+ * filter can be built (e.g. the commit-time field is not part of the read
schema), in which case
+ * the caller keeps the original read-time filter pushdown.
+ */
+ private def buildCommitTimeRowFilter(requiredFilters: Seq[Filter],
+ readSchema: StructType):
Option[JPredicate[InternalRow]] = {
+ if (requiredFilters.isEmpty) {
+ None
+ } else {
+
readSchema.getFieldIndex(HoodieRecord.COMMIT_TIME_METADATA_FIELD).flatMap { idx
=>
+ val allowedCommitTimes: Set[String] = requiredFilters.collect {
+ case In(attr, values) if attr ==
HoodieRecord.COMMIT_TIME_METADATA_FIELD =>
+ values.filter(_ != null).map(_.toString).toSet
+ }.flatten.toSet
+ if (allowedCommitTimes.isEmpty) {
+ None
+ } else {
+ Some(new JPredicate[InternalRow] {
+ override def test(row: InternalRow): Boolean =
+ !row.isNullAt(idx) &&
allowedCommitTimes.contains(row.getUTF8String(idx).toString)
Review Comment:
Done, the post-merge filter compares `UTF8String` values against a set built
once per file group.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala:
##########
@@ -353,6 +388,48 @@ class HoodieFileGroupReaderBasedFileFormat(tablePath:
String,
}
}
+ /**
+ * The incremental window end (latest commit time in the window), extracted
from the `In` filter on
+ * the commit-time metadata field in `requiredFilters`. Used to bound the
merge inputs so that log
+ * blocks committed after the window are not merged. Returns None when no
such filter is present.
+ */
+ private def incrementalWindowEnd(requiredFilters: Seq[Filter]):
Option[String] = {
+ val commitTimes: Seq[String] = requiredFilters.collect {
+ case In(attr, values) if attr == HoodieRecord.COMMIT_TIME_METADATA_FIELD
=>
+ values.filter(_ != null).map(_.toString).toSeq
+ }.flatten
+ if (commitTimes.isEmpty) None else Some(commitTimes.max)
+ }
+
+ /**
+ * Builds a predicate over merged rows that keeps only the records whose
`_hoodie_commit_time`
+ * falls within the incremental query window encoded by `requiredFilters`
(an `In` filter on the
+ * commit-time metadata field, produced by the incremental relations).
Returns None when no such
+ * filter can be built (e.g. the commit-time field is not part of the read
schema), in which case
+ * the caller keeps the original read-time filter pushdown.
+ */
+ private def buildCommitTimeRowFilter(requiredFilters: Seq[Filter],
+ readSchema: StructType):
Option[JPredicate[InternalRow]] = {
+ if (requiredFilters.isEmpty) {
+ None
+ } else {
+
readSchema.getFieldIndex(HoodieRecord.COMMIT_TIME_METADATA_FIELD).flatMap { idx
=>
+ val allowedCommitTimes: Set[String] = requiredFilters.collect {
Review Comment:
Done, `incrementalWindowCommitTimes` is now the only place that reads the
window commit times from the filters.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelationV2.scala:
##########
@@ -158,6 +158,76 @@ case class MergeOnReadIncrementalRelationV2(override val
sqlContext: SQLContext,
override def shouldIncludeLogFiles(): Boolean = fullTableScan
+ /**
+ * Collects the incremental file slices for the given modified partitions.
+ *
+ * Each returned file slice carries its base file plus all of its log files
so the
+ * [[org.apache.hudi.common.table.read.HoodieFileGroupReader]] can perform a
correct runtime merge.
+ * Resolving the current value of a record changed in the incremental window
requires the full file
+ * slice, not just the files written within the window:
+ * - EVENT_TIME_ORDERING: a record written in the window may carry a lower
ordering value than the
+ * version already in the base/earlier-log, so the existing version wins;
a window-only view
+ * cannot determine the winner.
+ * - Partial updates: a window log block holds only the changed columns, so
the unchanged columns
+ * must be filled in from the base file.
+ * (HUDI #18943.)
+ *
+ * The view is built from the (modified) partition listing (which includes
the latest base file of
+ * each file group, honoring the metadata table when enabled) and scoped
back to the file groups
+ * actually touched in the window (see [[affectedFileGroupIds]]) so
untouched file groups are not
+ * read. The commit-time record filter applied during the scan still
restricts the returned records
+ * to the incremental window.
+ *
+ * The view timeline is bounded to instants at or before `latestCommit` (the
window's last commit)
+ * so that base/log files written by later commits are not visible. Without
this bound a record
+ * updated again after the window would be merged with those later log files
and its merged commit
+ * time would fall outside the window, dropping the in-window change from
the result.
+ */
+ private def collectIncrementalFileSlices(partitionPaths: Seq[String],
latestCommit: String): Seq[FileSlice] = {
+ val engineContext = new HoodieSparkEngineContext(new
JavaSparkContext(sqlContext.sparkContext))
+ val fsView =
FileSystemViewManager.createInMemoryFileSystemViewWithTimeline(
+ engineContext, metaClient, fileIndex.metadataConfig,
timeline.findInstantsBeforeOrEquals(latestCommit))
+ try {
+ partitionPaths.flatMap { relativePartitionPath =>
+ fsView.getLatestMergedFileSlicesBeforeOrOn(relativePartitionPath,
latestCommit).iterator().asScala
Review Comment:
Done.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelationV2.scala:
##########
@@ -158,6 +158,76 @@ case class MergeOnReadIncrementalRelationV2(override val
sqlContext: SQLContext,
override def shouldIncludeLogFiles(): Boolean = fullTableScan
+ /**
+ * Collects the incremental file slices for the given modified partitions.
+ *
+ * Each returned file slice carries its base file plus all of its log files
so the
+ * [[org.apache.hudi.common.table.read.HoodieFileGroupReader]] can perform a
correct runtime merge.
+ * Resolving the current value of a record changed in the incremental window
requires the full file
+ * slice, not just the files written within the window:
+ * - EVENT_TIME_ORDERING: a record written in the window may carry a lower
ordering value than the
+ * version already in the base/earlier-log, so the existing version wins;
a window-only view
+ * cannot determine the winner.
+ * - Partial updates: a window log block holds only the changed columns, so
the unchanged columns
+ * must be filled in from the base file.
+ * (HUDI #18943.)
+ *
+ * The view is built from the (modified) partition listing (which includes
the latest base file of
+ * each file group, honoring the metadata table when enabled) and scoped
back to the file groups
+ * actually touched in the window (see [[affectedFileGroupIds]]) so
untouched file groups are not
+ * read. The commit-time record filter applied during the scan still
restricts the returned records
+ * to the incremental window.
+ *
+ * The view timeline is bounded to instants at or before `latestCommit` (the
window's last commit)
+ * so that base/log files written by later commits are not visible. Without
this bound a record
+ * updated again after the window would be merged with those later log files
and its merged commit
+ * time would fall outside the window, dropping the in-window change from
the result.
+ */
+ private def collectIncrementalFileSlices(partitionPaths: Seq[String],
latestCommit: String): Seq[FileSlice] = {
+ val engineContext = new HoodieSparkEngineContext(new
JavaSparkContext(sqlContext.sparkContext))
+ val fsView =
FileSystemViewManager.createInMemoryFileSystemViewWithTimeline(
+ engineContext, metaClient, fileIndex.metadataConfig,
timeline.findInstantsBeforeOrEquals(latestCommit))
+ try {
+ partitionPaths.flatMap { relativePartitionPath =>
+ fsView.getLatestMergedFileSlicesBeforeOrOn(relativePartitionPath,
latestCommit).iterator().asScala
+ .filter(fs => affectedFileGroupIds.contains(fs.getFileId))
+ }
+ } finally {
+ fsView.close()
+ }
+ }
+
+ /**
+ * Builds the incremental file slices directly from the files recorded in
the window's commits
+ * (`affectedFilesInCommits`), as the read path did before HUDI #18943.
+ *
+ * This is used only when [[hasMissingAffectedFiles]] is true and the
full-table-scan fallback is
+ * disabled: some files referenced by the window have been removed (e.g. by
cleaning), so there is
+ * no correct incremental result to produce. Listing from the recorded files
(which include the
+ * missing paths) preserves the prior fail-early contract - the scan
surfaces a file-not-found
+ * error pointing the user at
`hoodie.datasource.read.incr.fallback.fulltablescan.enable` - instead
+ * of silently returning an empty/partial result from a fresh listing that
no longer sees those
+ * files.
+ */
+ private def legacyAffectedFileSlices(partitionPaths: Seq[String],
latestCommit: String): Seq[FileSlice] = {
Review Comment:
Done, both views now go through `latestMergedFileSlices`, which owns the
`close()`.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelationV2.scala:
##########
@@ -194,8 +270,7 @@ trait HoodieIncrementalRelationV2Trait extends
HoodieBaseRelation {
val fallbackToFullTableScan =
optParams.getOrElse(DataSourceReadOptions.INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN.key,
DataSourceReadOptions.INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN.defaultValue).toBoolean
- fallbackToFullTableScan && (startInstantArchived
- || affectedFilesInCommits.asScala.exists(fileStatus =>
!metaClient.getStorage.exists(fileStatus.getPath)))
+ fallbackToFullTableScan && (startInstantArchived ||
hasMissingAffectedFiles)
Review Comment:
Addressed, the listing no longer checks file existence up front; it only
lists partitions of slices without a base file in the window, and checks
existence only for those slices when their logs were compacted away.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelationV2.scala:
##########
@@ -158,6 +158,76 @@ case class MergeOnReadIncrementalRelationV2(override val
sqlContext: SQLContext,
override def shouldIncludeLogFiles(): Boolean = fullTableScan
+ /**
+ * Collects the incremental file slices for the given modified partitions.
+ *
+ * Each returned file slice carries its base file plus all of its log files
so the
+ * [[org.apache.hudi.common.table.read.HoodieFileGroupReader]] can perform a
correct runtime merge.
+ * Resolving the current value of a record changed in the incremental window
requires the full file
+ * slice, not just the files written within the window:
+ * - EVENT_TIME_ORDERING: a record written in the window may carry a lower
ordering value than the
+ * version already in the base/earlier-log, so the existing version wins;
a window-only view
+ * cannot determine the winner.
+ * - Partial updates: a window log block holds only the changed columns, so
the unchanged columns
+ * must be filled in from the base file.
+ * (HUDI #18943.)
+ *
+ * The view is built from the (modified) partition listing (which includes
the latest base file of
+ * each file group, honoring the metadata table when enabled) and scoped
back to the file groups
+ * actually touched in the window (see [[affectedFileGroupIds]]) so
untouched file groups are not
+ * read. The commit-time record filter applied during the scan still
restricts the returned records
+ * to the incremental window.
+ *
+ * The view timeline is bounded to instants at or before `latestCommit` (the
window's last commit)
+ * so that base/log files written by later commits are not visible. Without
this bound a record
+ * updated again after the window would be merged with those later log files
and its merged commit
+ * time would fall outside the window, dropping the in-window change from
the result.
+ */
Review Comment:
Done, added a case with two file groups in one partition that asserts the
listing returns only the touched one, with its base file attached.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]