PHILO-HE commented on PR #9993: URL: https://github.com/apache/incubator-gluten/pull/9993#issuecomment-2987406155
> > As you mentioned, `ManifestCommitter` is available in Hadoop 3.3.6, but you only changed it for spark-3.5. It seems introducing `ManifestCommitter` will cause compilation failure when building Gluten for other spark versions? > > Hello @PHILO-HE yes you are right, it will cause compilation failures when building Gluten for lower spark versions. Can we consider to upgrade the hadoop version to 3.3.6 for other spark profiles as well? @RushabhK, I think it may not be a feasible way to upgrade hadoop in Gluten. You may note that hadoop dependencies are in provided scope. Therefore, even if we do the upgrading in Gluten, we cannot guarantee the imported class is available at runtime. Right? Have you investigated how Spark avoids such issue without using this hadoop class? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
