xushiyan commented on code in PR #8876:
URL: https://github.com/apache/hudi/pull/8876#discussion_r1225425413
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala:
##########
@@ -60,11 +60,14 @@ case class MergeOnReadIncrementalRelation(override val
sqlContext: SQLContext,
override protected def timeline: HoodieTimeline = {
if (fullTableScan) {
- metaClient.getCommitsAndCompactionTimeline
+ filterTimelineForIncrementalQueryIfNeeded(metaClient,
Review Comment:
throwing an exception is a way to alert users so that they do not blindly
query again with the latest timestamp under the impression that all data up to
date was return, as we do not return cursor from in the incremental query
result. We only do this when not using transition time.
> we can recommend users to turn on use-transition-time if multi-writer is
enabled both for incremental query & streaming read
that's the intention. we can recommend users for configuring it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
