xushiyan commented on code in PR #8876:
URL: https://github.com/apache/hudi/pull/8876#discussion_r1221215132
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala:
##########
@@ -60,11 +60,14 @@ case class MergeOnReadIncrementalRelation(override val
sqlContext: SQLContext,
override protected def timeline: HoodieTimeline = {
if (fullTableScan) {
- metaClient.getCommitsAndCompactionTimeline
+ filterTimelineForIncrementalQueryIfNeeded(metaClient,
Review Comment:
from offline discussion with @danny0405 and @codope : the behavior should
be, for batch incremental query when we detect such in-between inflight
commits, we throw an exception to alert user, regardless of user providing end
timestamp or not. For streaming read, we recommend users to turn on use
transition time if multi-writer is enabled. Changes to include transition time
in the streaming checkpoint will be deferred for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
