> On 8月 2, 2016, 1:58 p.m., Sebastian Toader wrote: > > ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerDefaultImpl.java, > > line 51 > > <https://reviews.apache.org/r/50653/diff/1/?file=1459056#file1459056line51> > > > > The 'X' stands for the ISO timezone representation. > > > > I think grok supports ISO timezone format as well. Can you confirm that > > grok patterns do not support iso 8061 timezone format? > > > > If it doesn't than go ahead and change for X to Z > > Masahiro Tanaka wrote: > Thank you for reviewing! I used [this > site](https://grokdebug.herokuapp.com/) to check if the pattern is able to > parse the log. > Please try pattern A and pattern B below. > > Pattern A: > ``` > # input > 2016-07-21T01:52:49.875+09, User(admin), RemoteIp(192.168.72.1), > Operation(Repository update), RequestType(PUT), > url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5), > ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo > id(HDP-2.5), Base > URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025) > > # pattern > (?m)^%{TIMESTAMP_ISO8601:logtime},%{SPACE}%{GREEDYDATA:log_message} > ``` > > Pattern B: > ``` > # input (timestamp is different from above, others are the same) > 2016-07-21T01:52:49.875+0900, User(admin), RemoteIp(192.168.72.1), > Operation(Repository update), RequestType(PUT), > url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5), > ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo > id(HDP-2.5), Base > URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025) > > # pattern > (?m)^%{TIMESTAMP_ISO8601:logtime},%{SPACE}%{GREEDYDATA:log_message} > ``` > > Sebastian Toader wrote: > Can you try ```return new > SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS**XX**");```? According to the > documentation is a single pattern letter is used than any fraction of an hour > is ignored. For example, if the pattern is "X" and the time zone is > "GMT+05:30", "+05" is produced. > > Sebastian Toader wrote: > I meant: SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSXX") > > Masahiro Tanaka wrote: > I've tried that. The result (`/var/log/ambari-server/ambari-audit.log` is > like as below > ``` > 2016-08-02T21:57:58.799+0900, User(admin), RemoteIp(192.168.72.1), > Operation(Repository update), RequestType(PUT), > url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/redhat6/repositories/HDP-UTILS-1.1.0.21), > ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(redhat6), Repo > id(HDP-UTILS-1.1.0.21), Base > URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos6) > 2016-08-02T21:57:58.810+0900, User(admin), RemoteIp(192.168.72.1), > Operation(Repository update), RequestType(PUT), > url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/redhat7/repositories/HDP-UTILS-1.1.0.21), > ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(redhat7), Repo > id(HDP-UTILS-1.1.0.21), Base > URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7) > 2016-08-02T21:57:58.816+0900, User(admin), RemoteIp(192.168.72.1), > Operation(Repository update), RequestType(PUT), > url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/debian7/repositories/HDP-UTILS-1.1.0.21), > ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(debian7), Repo > id(HDP-UTILS-1.1.0.21), Base > URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/debian7) > 2016-08-02T21:57:58.823+0900, User(admin), RemoteIp(192.168.72.1), > Operation(Repository update), RequestType(PUT), > url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/redhat6/repositories/HDP-2.5), > ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(redhat6), Repo > id(HDP-2.5), Base > URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/centos6/2.x/BUILDS/2.5.0.0-1116) > ``` > > I think this format is same as > `SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSZ");` at least in JST. > > Oliver Szabo wrote: > in SimpleDateFormat: > Z Time zone RFC 822 time zone -0800 > X Time zone ISO 8601 time zone -08; -0800; -08:00 > > It would make sense to use XX instead of Z, because in the grok pattern > we uses ISO 8601 format for parsing (to keep it clear what is the format here)
OK, I'll update the patch to use "XX". - Masahiro ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50653/#review144381 ----------------------------------------------------------- On 8月 1, 2016, 7:23 p.m., Masahiro Tanaka wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/50653/ > ----------------------------------------------------------- > > (Updated 8月 1, 2016, 7:23 p.m.) > > > Review request for Ambari, Daniel Gergely, Dmytro Sen, Oliver Szabo, and > Sebastian Toader. > > > Bugs: AMBARI-17308 > https://issues.apache.org/jira/browse/AMBARI-17308 > > > Repository: ambari > > > Description > ------- > > In logsearch_feeder service log, we got errors like below > ``` > 2016-06-20 15:28:09,368 ERROR file=ambari-audit.log > org.apache.ambari.logfeeder.mapper.MapperDate LogFeederUtil.java:356 - Error > applying date transformation. isEpoch=false, > dateFormat=yyyy-MM-dd'T'HH:mm:ss.SSSZ, value=2016-06-20T15:28:08.000. > mapClass=map_date, input=input:source=file, > path=/var/log/ambari-server/ambari-audit.log, fieldName=logtime. Messages > suppressed before: 2 > java.text.ParseException: Unparseable date: "2016-06-20T15:28:08.000" > at java.text.DateFormat.parse(DateFormat.java:366) > at > org.apache.ambari.logfeeder.mapper.MapperDate.apply(MapperDate.java:83) > at org.apache.ambari.logfeeder.filter.Filter.apply(Filter.java:154) > at > org.apache.ambari.logfeeder.filter.FilterGrok.applyMessage(FilterGrok.java:291) > at > org.apache.ambari.logfeeder.filter.FilterGrok.flush(FilterGrok.java:320) > at org.apache.ambari.logfeeder.input.Input.flush(Input.java:125) > at > org.apache.ambari.logfeeder.input.InputFile.processFile(InputFile.java:430) > at org.apache.ambari.logfeeder.input.InputFile.start(InputFile.java:260) > at org.apache.ambari.logfeeder.input.Input.run(Input.java:100) > at java.lang.Thread.run(Thread.java:745) > ``` > ambari-audit.log is like below > ``` > 2016-07-21T01:52:49.875+09, User(admin), RemoteIp(192.168.72.1), > Operation(Repository update), RequestType(PUT), > url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5), > ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo > id(HDP-2.5), Base > URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025) > 2016-07-21T01:52:49.905+09, User(admin), RemoteIp(192.168.72.1), > Operation(Repository update), RequestType(PUT), > url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu16/repositories/HDP-2.5), > ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu16), Repo > id(HDP-2.5), Base > URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu16/2.x/BUILDS/2.5.0.0-1025) > 2016-07-21T01:52:50.015+09, User(admin), RemoteIp(192.168.72.1), > Operation(Repository update), RequestType(PUT), > url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-UTILS-1.1.0.21), > ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo > id(HDP-UTILS-1.1.0.21), Base > URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/ubuntu14) > ``` > I think date format of the ambari-audit.log (2016-07-21T01:52:49.875+09) > should be like 2016-07-21T01:52:49.875+0900, since grok-pattern can't handle > 2016-07-21T01:52:49.875+09 format. > > > Diffs > ----- > > > ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerDefaultImpl.java > ea6524f > > ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/input.config-ambari.json.j2 > 5f751bf > > Diff: https://reviews.apache.org/r/50653/diff/ > > > Testing > ------- > > mvn clean test > > > Thanks, > > Masahiro Tanaka > >
