Re: [DISCUSS] Dropping support for elastic 2.x

2017-10-04 Thread Otto Fowler
If we break out the indexing from the hdfswriting, we could just have two different topologies to configure couldn’t we? On October 4, 2017 at 13:20:19, Michael Miklavcic ( michael.miklav...@gmail.com) wrote: The question comes back to the DISCUSS I opened the other day about upgrading ES. I

Re: [DISCUSS] Dropping support for elastic 2.x

2017-10-04 Thread Casey Stella
Regarding backwards compatibility at the code level, what are the pros/cons (outside of the obvious con that the transport client will be deprecated)? I guess what I'm trying to get at is what do we get in terms of functionality moving to a new backwards-incompatible transport client? A separate

Re: [DISCUSS] Dropping support for elastic 2.x

2017-10-04 Thread Michael Miklavcic
I should note that there's a difference between supporting INSTALLING multiple versions versus being able to manage them. On Wed, Oct 4, 2017 at 11:20 AM, Michael Miklavcic < michael.miklav...@gmail.com> wrote: > The question comes back to the DISCUSS I opened the other day about > upgrading ES.

Re: [DISCUSS] Dropping support for elastic 2.x

2017-10-04 Thread Michael Miklavcic
The question comes back to the DISCUSS I opened the other day about upgrading ES. I believe we could theoretically maintain backwards compatibility, but we'd have to keep the existing TransportClient. It's not deprecated yet, but it will be. Keeping the ability to manage ES 2.x and 5.x+ via Ambari

Re: [DISCUSS] Dropping support for elastic 2.x

2017-10-04 Thread Casey Stella
Ok, so, whoever does this ES work, we should ensure the upgrade path is at least spelled out in the Upgrade doc. This would also probably, IMO, necessitate a major version change in metron. On Wed, Oct 4, 2017 at 1:07 PM, Justin Leet wrote: > Forgot the link >

Re: [DISCUSS] Dropping support for elastic 2.x

2017-10-04 Thread Justin Leet
Forgot the link https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html On Wed, Oct 4, 2017 at 1:07 PM, Simon Elliston Ball < si...@simonellistonball.com> wrote: > The simplest option would probably be to upgrade the ES and then reindex > from the HDFS store.

Re: [DISCUSS] Dropping support for elastic 2.x

2017-10-04 Thread Justin Leet
ES should be upgradeable without wiping. It's the client itself that isn't backwards compatible. It'll require both an upgrade of Metron and an ES cluster. On Wed, Oct 4, 2017 at 1:05 PM, Casey Stella wrote: > So, how would this work in an upgrade scenario that does not

Re: [DISCUSS] Dropping support for elastic 2.x

2017-10-04 Thread Simon Elliston Ball
The simplest option would probably be to upgrade the ES and then reindex from the HDFS store. Alternatively there are means to do inplace upgrades from 2.x to 5.x I believe. Simon > On 4 Oct 2017, at 18:05, Casey Stella wrote: > > So, how would this work in an upgrade

Re: [DISCUSS] Dropping support for elastic 2.x

2017-10-04 Thread Casey Stella
So, how would this work in an upgrade scenario that does not involve losing the existing indexed data? On Wed, Oct 4, 2017 at 12:55 PM, Michael Miklavcic < michael.miklav...@gmail.com> wrote: > The client I'm currently working on moving towards would *not* be backwards > compatible. >

Re: [DISCUSS] Dropping support for elastic 2.x

2017-10-04 Thread Michael Miklavcic
The client I'm currently working on moving towards would *not* be backwards compatible. https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-compatibility.html " The High Level Client is guaranteed to be able to communicate with any Elasticsearch node running on

[DISCUSS] Dropping support for elastic 2.x

2017-10-04 Thread Simon Elliston Ball
A number of people are currently working on upgrading the ES support in Metron to 5.x (including the clients, and the mpack managed install). Would anyone have any objections to dropping formal support for 2.x as a result of this work? In theory the clients should be backward compatible against