HeartSaVioR commented on pull request #28841: URL: https://github.com/apache/spark/pull/28841#issuecomment-652857478
Your math is correct and I agree it helps. The thing is, how much it would help? When we talk about streaming we are probably talking about the query which runs months. The lower bound is static and the query will get boosted on the earlier batch to filter out older files, but once the query catches up, situation would be similar. I agree the overall complication is not a goal for this PR - just wanted to allow me think less when I go through such complication. One more option, one more complication. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
