Hi Simon, We have temporarily decided to just change it with "_" for HDFS to avoid all the headaches of the bugs and issues that can be raised by using unsupported separators for ORC/Hive and Spark. However, I am not quite confident with "_" as an option for the community as it becomes similar to normal Metron separator. Maybe it would be nice to have an ability to change the separator to any other character and let users decide what they want to use.
Cheers, Ali On Tue, Aug 14, 2018 at 12:14 AM Simon Elliston Ball < si...@simonellistonball.com> wrote: > Do you have any suggestions for what would make sense as a delimiter? > > On 9 August 2018 at 05:57, Ali Nazemian <alinazem...@gmail.com> wrote: > > > Hi All, > > > > I was wondering if we can change the field separators in Metron to be > able > > to make it Hive/ORC friendly. I could find the following PR, but neither > > dot nor colon is very Hive and ORC friendly and they will cause some > > issues. Hence, I wanted to see if it is possible to change the field > > separator to something else or even give users an ability to define what > > separator to be used to make the data model consistent across > Elasticsearch > > and HDFS. > > > > https://github.com/apache/metron/pull/1022 > > > > Cheers, > > Ali > > > > > > -- > -- > simon elliston ball > @sireb > -- A.Nazemian
