> On Aug. 2, 2016, 1:58 p.m., Sebastian Toader wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerDefaultImpl.java,
> >  line 51
> > <https://reviews.apache.org/r/50653/diff/1/?file=1459056#file1459056line51>
> >
> >     The 'X' stands for the ISO timezone representation. 
> >     
> >     I think grok supports ISO timezone format as well. Can you confirm that 
> > grok patterns do not support iso 8061 timezone format?
> >     
> >     If it doesn't than go ahead and change for X to Z
> 
> Masahiro Tanaka wrote:
>     Thank you for reviewing! I used [this 
> site](https://grokdebug.herokuapp.com/) to check if the pattern is able to 
> parse the log.
>     Please try pattern A and pattern B below.
>     
>     Pattern A:
>     ```
>     # input
>     2016-07-21T01:52:49.875+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
>     
>     # pattern
>     (?m)^%{TIMESTAMP_ISO8601:logtime},%{SPACE}%{GREEDYDATA:log_message}
>     ```
>     
>     Pattern B:
>     ```
>     # input (timestamp is different from above, others are the same)
>     2016-07-21T01:52:49.875+0900, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
>     
>     # pattern
>     (?m)^%{TIMESTAMP_ISO8601:logtime},%{SPACE}%{GREEDYDATA:log_message}
>     ```
> 
> Sebastian Toader wrote:
>     Can you try ```return new 
> SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS**XX**");```? According to the 
> documentation is a single pattern letter is used than any fraction of an hour 
> is ignored. For example, if the pattern is "X" and the time zone is 
> "GMT+05:30", "+05" is produced.
> 
> Sebastian Toader wrote:
>     I meant: SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSXX")

I've tried that. The result (`/var/log/ambari-server/ambari-audit.log` is like 
as below
```
2016-08-02T21:57:58.799+0900, User(admin), RemoteIp(192.168.72.1), 
Operation(Repository update), RequestType(PUT), 
url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/redhat6/repositories/HDP-UTILS-1.1.0.21),
 ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(redhat6), Repo 
id(HDP-UTILS-1.1.0.21), Base 
URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos6)
2016-08-02T21:57:58.810+0900, User(admin), RemoteIp(192.168.72.1), 
Operation(Repository update), RequestType(PUT), 
url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/redhat7/repositories/HDP-UTILS-1.1.0.21),
 ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(redhat7), Repo 
id(HDP-UTILS-1.1.0.21), Base 
URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7)
2016-08-02T21:57:58.816+0900, User(admin), RemoteIp(192.168.72.1), 
Operation(Repository update), RequestType(PUT), 
url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/debian7/repositories/HDP-UTILS-1.1.0.21),
 ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(debian7), Repo 
id(HDP-UTILS-1.1.0.21), Base 
URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/debian7)
2016-08-02T21:57:58.823+0900, User(admin), RemoteIp(192.168.72.1), 
Operation(Repository update), RequestType(PUT), 
url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/redhat6/repositories/HDP-2.5),
 ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(redhat6), Repo 
id(HDP-2.5), Base 
URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/centos6/2.x/BUILDS/2.5.0.0-1116)
```

I think this format is same as 
`SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSZ");` at least in JST.


- Masahiro


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50653/#review144381
-----------------------------------------------------------


On Aug. 1, 2016, 7:23 p.m., Masahiro Tanaka wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50653/
> -----------------------------------------------------------
> 
> (Updated Aug. 1, 2016, 7:23 p.m.)
> 
> 
> Review request for Ambari, Daniel Gergely, Dmytro Sen, Oliver Szabo, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-17308
>     https://issues.apache.org/jira/browse/AMBARI-17308
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> In logsearch_feeder service log, we got errors like below
> ```
> 2016-06-20 15:28:09,368 ERROR file=ambari-audit.log 
> org.apache.ambari.logfeeder.mapper.MapperDate LogFeederUtil.java:356 - Error 
> applying date transformation. isEpoch=false, 
> dateFormat=yyyy-MM-dd'T'HH:mm:ss.SSSZ, value=2016-06-20T15:28:08.000. 
> mapClass=map_date, input=input:source=file, 
> path=/var/log/ambari-server/ambari-audit.log, fieldName=logtime. Messages 
> suppressed before: 2
> java.text.ParseException: Unparseable date: "2016-06-20T15:28:08.000"
>       at java.text.DateFormat.parse(DateFormat.java:366)
>       at 
> org.apache.ambari.logfeeder.mapper.MapperDate.apply(MapperDate.java:83)
>       at org.apache.ambari.logfeeder.filter.Filter.apply(Filter.java:154)
>       at 
> org.apache.ambari.logfeeder.filter.FilterGrok.applyMessage(FilterGrok.java:291)
>       at 
> org.apache.ambari.logfeeder.filter.FilterGrok.flush(FilterGrok.java:320)
>       at org.apache.ambari.logfeeder.input.Input.flush(Input.java:125)
>       at 
> org.apache.ambari.logfeeder.input.InputFile.processFile(InputFile.java:430)
>       at org.apache.ambari.logfeeder.input.InputFile.start(InputFile.java:260)
>       at org.apache.ambari.logfeeder.input.Input.run(Input.java:100)
>       at java.lang.Thread.run(Thread.java:745)
> ```
> ambari-audit.log is like below
> ```
> 2016-07-21T01:52:49.875+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
> 2016-07-21T01:52:49.905+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu16/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu16), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu16/2.x/BUILDS/2.5.0.0-1025)
> 2016-07-21T01:52:50.015+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-UTILS-1.1.0.21),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-UTILS-1.1.0.21), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/ubuntu14)
> ```
> I think date format of the ambari-audit.log (2016-07-21T01:52:49.875+09) 
> should be like 2016-07-21T01:52:49.875+0900, since grok-pattern can't handle 
> 2016-07-21T01:52:49.875+09 format.
> 
> 
> Diffs
> -----
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerDefaultImpl.java
>  ea6524f 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/input.config-ambari.json.j2
>  5f751bf 
> 
> Diff: https://reviews.apache.org/r/50653/diff/
> 
> 
> Testing
> -------
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Masahiro Tanaka
> 
>

Reply via email to