[
https://issues.apache.org/jira/browse/DRILL-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16054985#comment-16054985
]
Kunal Khatua commented on DRILL-4955:
-------------------------------------
[~cgivre] is there a PR for this?
I've experimented with LogStash / GrokParser and it did a pretty decent job.
That would probably be more powerful for adopting log parsing (incl. Drill).
Personally, IMHO, it would be neat if we could use Drill to parse and analyze
its own logs, but I didn't pursue this much further with the ELK combo after a
point becauseI felt that a good design would be to have the logs itself exposed
as a System table.
> Log Parser for Drill
> --------------------
>
> Key: DRILL-4955
> URL: https://issues.apache.org/jira/browse/DRILL-4955
> Project: Apache Drill
> Issue Type: New Feature
> Components: Storage - Text & CSV
> Affects Versions: 1.9.0
> Reporter: Charles Givre
> Labels: features
> Fix For: Future
>
>
> I've been experimenting with a generic log parser for Drill. The basic
> concept is that if you wanted Drill to ingest log files such as this MySQL
> log:
> {code}
> 070823 21:00:32 1 Connect root@localhost on test1
> 070823 21:00:48 1 Query show tables
> 070823 21:00:56 1 Query select * from category
> 070917 16:29:01 21 Query select * from location
> 070917 16:29:12 21 Query select * from location where id = 1 LIMIT
> 1
> {code}
> You probably could do it with the various string manipulation methods such as
> split, substring etc. but you'd end up with some ugly and very complex
> queries.
> The extension I've built allows you to supply Drill with a regex for the
> formatting and a list of fields as shown below.
> {code}
> "log": {
> "type": "log",
> "extensions": [
> "log"
> ],
> "fieldNames": [
> "date",
> "time",
> "pid",
> "action",
> "query"
> ],
> "pattern":
> "(\\d{6})\\s(\\d{2}:\\d{2}:\\d{2})\\s+(\\d+)\\s(\\w+)\\s+(.+)"
> }
> {code}
> You can then query this log files in this format in Drill. I'd like to
> submit this for inclusion in Drill if there is interest.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
