dbatomic commented on PR #46247: URL: https://github.com/apache/spark/pull/46247#issuecomment-2110406204
> Let's make clear the scope of tests we are adding here. I see the PR title is about "stateless" but you are also aware that deduplication is "stateful". While I agree that we probably won't want to add the collation test for all stateful operators, let's make the scope more clear in PR title. > Let's make clear the scope of tests we are adding here. I see the PR title is about "stateless" but you are also aware that deduplication is "stateful". While I agree that we probably won't want to add the collation test for all stateful operators, let's make the scope more clear in PR title. > Let's make clear the scope of tests we are adding here. I see the PR title is about "stateless" but you are also aware that deduplication is "stateful". While I agree that we probably won't want to add the collation test for all stateful operators, let's make the scope more clear in PR title. Right, I updated both PR title and PR description. And yes, tests for collations are still pretty ad-hoc/selective. Goal of this PR is to assert that basics work. As we create more thorough plan for collations and streaming we will start adding better organized test strategies. Let me know if you think now is a good time to start with this. I was also thinking about creating new test suite only for collations, but that seemed like an overkill for this change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
