szehon-ho commented on PR #41683: URL: https://github.com/apache/spark/pull/41683#issuecomment-1675271802
Hm, in the original implementation (https://github.com/apache/spark/pull/41683/commits/1b63105ee4b62e1e582e8e4c84635fc0c39b1337), it did not differentiate and I made an UnresolvedRelation with options. That's the same as the DataStreamReader API. Do we want to differentiate? Actually after changing first argument to use TABLE(), because ResolveFunctions on TVF expect the children to all be resolved (1), looks like this function gets ResolvedRelation as argument. DSV2Relation has options, but StreamingDSV2Relation does not have options, so options then only applies to the former. I am contemplating reverting back to the original implementation though, and not take TABLE(). It would be nice to pass options into the relation-resolution process itself, any thoughts on that? [1]: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala#L2085) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
