rdblue commented on issue #24129: [SPARK-27190][SQL] add table capability for streaming URL: https://github.com/apache/spark/pull/24129#issuecomment-481919141 > Even if the check is `supportsAny(MICROBATCH_READ, CONTINUOUS_READ)`, something reason able needs to be added to the analysis checks. I still think that this needs to do some checks in the analyzer. I don't like having validation rules in so many places across the codebase because I think it isn't maintainable. Quite a few things can be caught in the analyzer, if I understand correctly. I know that the streaming mode isn't known at planning time, but that doesn't affect the ability to check the write side because there isn't a difference between the modes in the write API. We can also add a check to verify that all of the sources support streaming reads and streaming writes and there aren't any batch-only data sources (if that's not allowed). I'd also like to see a check that at least one streaming mode is compatible with all sources and sinks. That way, if one source supports only continuous and another supports only micro-batch, the analyzer catches it.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
