pedrumj2 commented on PR #13639: URL: https://github.com/apache/gluten/pull/13639#issuecomment-6900943914
> @pedrumj2, thanks for the PR! > > Could we adapt the test checker instead? On Spark 4.x, all length checks go through `assertLengthCheckFailure`, so the Gluten suite can override it to accept any exception whose cause chain contains Velox's `Exceeds allowed length limitation: <n>`. Then the copied 4.x tests can be removed. On Spark 3.4 / 3.5, the checks are inline in each test, so I think we can keep the existing Gluten copies there. Since Spark 3.4 support will be removed soon (#13573), that leaves only the Spark 3.5 copies, which should be fine to keep. > > The point is that a different exception type or message should be fine for users, as long as the query fails for the same reason and the message clearly explains it. Does this make sense? Thanks for the review @philo-he > Could we adapt the test checker instead? On Spark 4.x, all length checks go through `assertLengthCheckFailure`, so the Gluten suite can override it to accept any exception whose cause chain contains Velox's `Exceeds allowed length limitation: <n>` I can definitely follow with using the test checker. However I was thinking a bit more broadly how we can solve this test duplication overall. Seems to be common pattern. My only concern is that with this approach we would rely on the spark repo to provide us with these extensions so we can leverage them in our tests. Please see this [comment](https://github.com/apache/spark/pull/52996#pullrequestreview-3449022354) while they do add these helper functions from time to time but it seems they are reluctant to provide them consistently, meaning I'm not sure we can solve this for all our tests. > The point is that a different exception type or message should be fine for users, as long as the query fails for the same reason and the message clearly explains it. Does this make sense? I agree, this definitely make sense. There are two options for raising exceptions each have their pros and cons: - Have a gluten exception type with a different user facing message than Spark - Maintain same exception with Spark wrapped in a gluten exception Curious I looked through the repo but wasn't able to find a discussion on which direction the repo prefers to take. Do you think it would be worthwhile to have a broader discussion on this? I opened an issue here https://github.com/apache/gluten/issues/13647 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
