dwsmith1983 commented on PR #5872: URL: https://github.com/apache/datafusion-comet/pull/5872#issuecomment-6146523063
@andygrove @sunchao @parthchandra which `hadoop-aws` version would you like the spark-4.2 profile to use? Spark 4.0.4 ships Hadoop 3.4.1, 4.1.3 ships 3.4.2, and 4.2.0 ships 3.5.0. This PR moves the default (Spark 4.1) to 3.4.2 and pins spark-4.0 and spark-4.2 to 3.4.1, the version both had before. For spark-4.2 that leaves `hadoop-aws` a minor line behind the Hadoop the profile runs on, while `hadoop-azure` in the same profile is already 3.5.0. The profile provider adapter tests also skip there, since they need 3.4.2 or later. Options for spark-4.2: 1. Keep 3.4.1 and give the reason in the pom comment. Nothing changes for that profile, and the profile provider tests stay skipped on it. 2. Drop the pin so it inherits 3.4.2. The tests run on 4.2, one line behind Spark. 3. Pin 3.5.0, matching Spark 4.2 and `hadoop-azure`. `hadoop-aws` 3.5.0 is built against AWS SDK v2 2.35.4, while Comet pins 2.31.51 for every profile, so this needs either a check that the adapters work on the older SDK or an SDK bump for that profile. Option 3 tests what Spark 4.2 users actually run. It could also go in its own PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
