xkrogen commented on pull request #30891:
URL: https://github.com/apache/spark/pull/30891#issuecomment-755449849
That assumes no other profile changes are necessary to run against 3.2 vs.
3.3. For example, here is a snapshot of the 2.6 and 2.7 profiles from 2018:
```
<profile>
<id>hadoop-2.6</id>
<!-- Default hadoop profile. Uses global properties. -->
</profile>
<profile>
<id>hadoop-2.7</id>
<properties>
<hadoop.version>2.7.3</hadoop.version>
<curator.version>2.7.1</curator.version>
</properties>
</profile>
```
In this case, using `-Phadoop-2.7 -Dhadoop.version=2.6.1` would be
insufficient since there are other profile changes necessary (the curator
version).
Back in 2015 the situation required even more tweaking:
```
<profile>
<id>hadoop-2.2</id>
<!-- SPARK-7249: Default hadoop profile. Uses global properties. -->
</profile>
<profile>
<id>hadoop-2.3</id>
<properties>
<hadoop.version>2.3.0</hadoop.version>
<jets3t.version>0.9.3</jets3t.version>
</properties>
</profile>
<profile>
<id>hadoop-2.4</id>
<properties>
<hadoop.version>2.4.0</hadoop.version>
<jets3t.version>0.9.3</jets3t.version>
</properties>
</profile>
<profile>
<id>hadoop-2.6</id>
<properties>
<hadoop.version>2.6.0</hadoop.version>
<jets3t.version>0.9.3</jets3t.version>
<zookeeper.version>3.4.6</zookeeper.version>
<curator.version>2.6.0</curator.version>
</properties>
</profile>
```
This is why I bring up the compatibility of Hadoop these days. If we squash
it down to a single `hadoop-3` profile, this makes the assumption that all
Hadoop versions in the 3.X line will have completely compatible dependency
trees.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
