vladislav-sidorovich commented on code in PR #15407: URL: https://github.com/apache/iceberg/pull/15407#discussion_r4270274838
########## dev/.rat-excludes: ########## @@ -29,4 +29,5 @@ sitemap.xml .python-version **/*_index.md **/.venv/** +**/_delta_log/_last_checkpoint Review Comment: > we are potentially packaging _delta_log/ in the source distribution. ... No, it's only test usage. But it's good point, I need to clean-up this file. > I have some concerns with adding 100+ "golden files" into the repo, is it really necessary? can we just generate these? It's trade-off `Static Golden Files` vs. `Dynamic Generation`. From my point of view, static files are more reliable for testing scenarios. Some motivation behind my point of view: 1. The purpose of these golden files is to test protocol conformance across specific. So, it's exactly the case. 2. Since protocol specs are immutable, these files shouldn't change over time. 3. Generating all of these tables dynamically during test runs requires spinning up a full writer engine (like Spark + `delta-spark`) in the middle of unit tests. That adds an extra layer of tooling that can be error-prone. 4. This follows the exact same philosophy Iceberg recently adopted with [`apache/iceberg-verification`](https://github.com/apache/iceberg-verification), where language-neutral, immutable conformance fixtures are used to verify reader implementations rather than having each test generate its own data on the fly. Unfortunately we can't pull the golden tables as a test-scoped Maven dependency/archive during the build, IMHO it would be an ideal solution. @anoopj what do you think, is it possible to create an artifact with the golden tables? What are your thoughts on this? Could you share a bit more about your concern so we can find the best approach? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
