To put things in perspective about the upgrade concern and libraries. It is very often the case that when using elastic's hadoop library, supporting most recent Elasticsearch requires immediate upgrade. This means a release of the job.
That part is not so much the problem expect in the issue mentioned above doing that for ES 7 (which we are nagged constantly about) breaks 2.x and also 5.x. To support the few on 7 will break probably a lot.. who knows how many.. on 5. Ideally ES will have integration tests like we do to know when things break but this has not occurred yet and literally the answer I was given was that the policy will be current major and last minor. IOTW not immediately upgrading to ES libraries will have an effect which is more angry people like we have now spamming our issues list about most recent ES version. At least the main repo we are insulated as we have our own client library, so we dont break as often as upstream. However, dependencies suddenly not working will be a problem. To your point though.. we could make a matrix on the last version you can use for what ES version. Bugs in the dependencies jobs are less critical than the server. So, maybe this is the best path for now? Keep a matrix on the dependencies (spark) job? -A
