Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5786#issuecomment-101458849
We all understand that POMs define how assemblies are built. This can't
have nothing to do with them.
As I say, the build works fine in, say, local mode (well, no obvious
problems). Jenkins tests are happy since they don't rely on defaults. It's not
true that nothing works, but there's a problem. This is narrowly about the
Hadoop user's perilous expectations of defaults.
I don't expect the default assembly to work on a Hadoop 1.x cluster, but
it's not supposed to now in Spark 1.4. You're suggesting reverting to restore
the default to work with Hadoop 1.x, but then that trips a different
version-related problem: the published POM for Spark 1.3 already references
Hadoop 2.2.0. "Fixing" that may make the default assembly work for Hadoop 1.x
again as it did in Spark 1.2, but then it yet again changes the transitive deps
of anyone relying on Spark Core artifacts in Maven. This is why I don't think
reverting to `hadoop.version=1.0.4` is a good solution, and maybe that is the
only point still being batted around.
But Spark 1.4 is in a no-mans-land where the defaults don't work on 1.x
(expected) and apparently don't quite work on 2.x (not expected). You'd think
that at least one does. That's plainly suboptimal, and while not a
show-stopper, needs fixing. I don't think anyone disputes that this PR would do
the trick. Further, I like the idea of encouraging people to do the right
thing, what the release has always safely done: specify Hadoop profile when it
matters.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
