wmoustafa commented on PR #4925: URL: https://github.com/apache/iceberg/pull/4925#issuecomment-1253381372
@amogh-jahagirdar very good points. Following this line of thought, we should probably also address the question of whether the SQL text should be validated in Iceberg/the API implementation, or validation is left to the enclosing engine. If it is validated in Iceberg/the API implementation, then Iceberg/implementation needs to understand the dialects as well. If it is left to the enclosing engine, there is the risk of passing an invalid view text. Similarly, whether it is responsibility of Iceberg (or the API implementation) to ensure the semantic equivalence of all the representations (and not only the schemas). Finally, if we assume there is a translation tool, we may consider if it makes sense to store one representation and let the remaining representations be handled by translation, as opposed to letting the caller generate all the representations, which as you indicated, could fall out of sync. By the way, I believe adding new representations (case #2) should not increment the view version. To answer your question about Coral schema validation, Coral infers the schema of a view given the view text and base table schemas (see some [test cases](https://github.com/linkedin/coral/blob/ceb8fdfc8c91eb58a0864ca92c22361361f73c3c/coral-schema/src/test/java/com/linkedin/coral/schema/avro/ViewToAvroSchemaConverterTests.java#L24)). It uses Avro schemas for representing the schemas (due to its rich semantics), but extension to use Iceberg schemas is straightforward. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
