yashmayya commented on PR #13193: URL: https://github.com/apache/pinot/pull/13193#issuecomment-2133432371
> For example testHardCodedQueries has a larger of set of queries Is this what you're referring to - https://github.com/apache/pinot/blob/518fd180981ffa614042b8a5236650a2af41bfe5/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTestSet.java#L287? If yes, that's from the integration test suite and is completely separate from the backward compatibility regression test framework (added in https://github.com/apache/pinot/pull/6129). The latter is a set of bash scripts and yaml configurations to set up controllers, brokers, servers and upgrading / downgrading each sequentially while also ingesting real-time / offline data and running a bunch of static queries. > I am curious what the effort will be to increase coverage of different query plans. I think it should be pretty straightforward to add more datasets and queries once we have this baseline in place. It's currently quite important that we at least have _some_ backward compatibility regression tests for the multi-stage query engine since there's none currently and the only way to detect that a commit is breaking backward compatibility with it is via either eyeballing the code or manually setting up a cluster for an upgrade test. The compatibility verifier has a [script to generate test data](https://github.com/apache/pinot/blob/master/compatibility-verifier/sample-test-suite/config/dataGenerator.py) but nothing to generate random queries though. > Another question for the future. The concept of plan versions has to be introduced. Compatibility of a specific version has to be tested ? I'm not sure I follow? Currently, this PR adds a new GitHub Actions workflow that can be manually triggered during Pinot releases to verify that backward compatibility for the multi-stage query engine is maintained with previously released versions. This also adds a new check that runs on every PR to ensure that the PR isn't introducing a backward incompatible change for the multi-stage query engine (currently only run versus `master` since we already know that there were some backward incompatible changes made previously). In the future, this will be updated to also run against newer releases since we want to maintain compatibility going forward. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
