Integrity checks across both, in case there was a concern of tampering on either side is my first thought. Being able to bubble something up into a hash comparison would be nice, without needing to transform (although difficult due to the different formats inherent in storage). It could also be confusing if fields changed part way through the currently retained data for your transforms. It could also make it easier to write a UI that queries across all possible indexes.
That said, I'm completely comfortable taking the previously suggested approach and I'm already working on a PR assuming it. Jon On Tue, May 9, 2017 at 11:03 AM Simon Elliston Ball < si...@simonellistonball.com> wrote: > Jon, > > What would you use case be for comparison? Reconciliation of the sources? > In theory both should be identical since they’re indexed from the same > source. There should never be any reason to combine ES and HDFS indexing, > unless there is a use case I’m missing... > > Simon > > > > On 9 May 2017, at 15:00, zeo...@gmail.com <zeo...@gmail.com> wrote: > > > > Have we ever considered the use case where we might want to compare data > > across our stores, like ES and HDFS? I think there is some merit in > > keeping the keys consistent across them (and doing transforms earlier in > > the stack rather than later). > > > > Jon > > > > On Tue, May 9, 2017 at 9:53 AM Otto Fowler <ottobackwa...@gmail.com> > wrote: > > > >> ElasticsearchFieldNameConverter.java > >> > >> > >> > >> On May 9, 2017 at 09:46:09, Nick Allen (n...@nickallen.org) wrote: > >> > >> The ES indexer replaces periods, I believe. Are you seeing periods hit > >> Elasticsearch? > >> > >> And architecturally that kind of logic should be done in the indexers > >> anyways. > >> > >> On Tue, May 9, 2017 at 9:41 AM, zeo...@gmail.com <zeo...@gmail.com> > wrote: > >> > >>> Is there a reason why the bro parser allows periods > >>> <https://github.com/apache/incubator-metron/blob/master/ > >>> metron-platform/metron-parsers/src/main/java/org/ > >>> apache/metron/parsers/bro/JSONCleaner.java#L56> > >>> in the keys if we can't index it (ES 2.3.3 does not allow periods in > >>> indexes)? Would anybody be opposed to me modifying the bro parser to > >>> rewrite "."s to "_"s for the short term, until we get ES upgraded > >>> <https://issues.apache.org/jira/browse/METRON-939>? > >>> > >>> Jon > >>> -- > >>> > >>> Jon > >>> > >> > > -- > > > > Jon > > -- Jon
