boneanxs commented on code in PR #8876:
URL: https://github.com/apache/hudi/pull/8876#discussion_r1222386708
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala:
##########
@@ -60,11 +60,14 @@ case class MergeOnReadIncrementalRelation(override val
sqlContext: SQLContext,
override protected def timeline: HoodieTimeline = {
if (fullTableScan) {
- metaClient.getCommitsAndCompactionTimeline
+ filterTimelineForIncrementalQueryIfNeeded(metaClient,
Review Comment:
If user doesn't provide the end timestamp, we still throw an exception if
finding hollow instants? How abt setting the firstIncompleteCommit as end
timestamp?
Throw exceptions disrupts our concurrency control to some extent(In the
subconscious of many users, reader side won't be affected if using multi
writers for hudi), but for now it looks we don't have other easy workarounds. I
think we can recommend users to turn on use-transition-time if multi-writer is
enabled both for incremental query & streaming read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
