+1 for the feature If possible may be you could include grok format for specifying log structure. https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
- Tushar. On Thu, Nov 17, 2016 at 11:48 PM, Sanjay Pujare <[email protected]> wrote: > +1, I like this feature. > > On 11/17/16, 3:26 AM, "Shraddha Jog" <[email protected]> wrote: > > Dear community, > > We would like to add operator in malhar for parsing different types of > logs. > Idea of this operator is to read log data records of known formats such as > Syslog, common log, combined log, extended log etc from the upstream in a > DAG, parse/validate it based on the configured format and emit the > validated POJO to the downstream. > > We are not targeting log formats from particular library as such but the > default formats for common log, combined log, extended log and sys log. > Also if user has some specific log format then that could be provided in a > property and operator will parse the log according to the given format. > > Properties: > LogFileFormat : Property to define data format for the log data record > being read at the Input Port. It can be either from the above four default > log formats or a json specifying fields and regular expression. More > details can be found in the document. > > Ports : > 1. ParsedLog: This port shall emit the parsed/validated POJO object > created > based on the log format configured by the user. > > 2. ErrorPort: This port shall emit the error log data record. > > Proposed design can be found here > > <https://docs.google.com/document/d/1RoTOUx_0chwTSahGxiIXlgACgRNfXiVv17hMezxFz74/edit?usp=sharing> > . > > Thanks, > Shraddha > > -- > This e-mail, including any attached files, may contain confidential and > privileged information for the sole use of the intended recipient. Any > review, use, distribution, or disclosure by others is strictly prohibited. > If you are not the intended recipient (or authorized to receive > information > for the intended recipient), please contact the sender by reply e-mail and > delete all copies of this message. > > > >
