cloud-fan commented on issue #24129: [SPARK-27190][SQL] add table capability for streaming URL: https://github.com/apache/spark/pull/24129#issuecomment-483484108 @rdblue `V2WriteSupportCheck` only checks write capability not scan. > For any streaming scan of a V2Relation, validate that the underlying table supports either micro-batch or continuous mode. This needs to create an unresolved streaming DS v2 relation, and ask `DataStreamReader` to create this unresolved relation, instead of checking the scan capability and creating a resolved relation. I don't agree that we HAVE TO do it in this PR while the PR that adds batch scan ability did not. > For any streaming write to a V2Relation, validate that the underlying table supports streaming writes. This needs to move the creation of streaming writing logical plan from `MicroBatchExecution`/`ContinuousExecution` to `DataStreamWriter`. The writing logical plan is different between micro-batch and continuous, we also need an unresolved writing plan, and resolve it when we know the streaming execution mode. This is a major refactor > For the set of streaming scans in the plan, validate that the intersection of supported modes is non-empty. For example, fail if s1 supports only microbatch and s2 supports only continuous. Same as the first point. I don't understand why you think this PR is incomplete. It adds streaming scan/write capability and checks them correctly. It's a different story to centralize the check, which can be done in a followup.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
