Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/12004
2.6 vs 2.7 vs later releases âa moving target, with AWS versions and
other issues to worry about.
[HADOOP-13687](https://issues.apache.org/jira/browse/HADOOP-13687) is going to
add a single thing to declare a dependency on, but even there, care with
transitive jackson dependencies
Sean: I'm proposing pulling out all the tests excluding some to check up
classpaths (ie. transitive dependency logic), which can run as junit; leaving a
patch of: transitive dependency management, docs, dependency checks.
Those tests need aws/azure credentials what I'd pull out. They run happily
jenkins, but cannot be run off ASF jenkins builds for security reasons: too
easy to steal credentials. You are however free to run them privately. The bits
in the `src/main` module are actually the underlying tests, designed to also be
runnable as spark-submit operations against real clusters; the entry points
take size parameters so you can scale them up. Essentially: foundational
integration tests that the installed cluster has the libraries, credentials and
network access. Again, if I move this standalone then they are something which
can be used in cluster tests and install-time smoke tests.
Regarding the feature branch comment, all I've done is PRs, in public
development of WiP coding, code where I always welcome co-developers and
reviews. Maybe I've misread it, but it sounds like I'm being encouraged to not
even do that. Which leaves me where?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
