Re: Review Request 50653: AMBARI-17308 Ambari Logfeeder outputs a lot of errors due to parse date

2016-08-04 Thread Masahiro Tanaka 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50653/
---

(Updated Aug. 4, 2016, 8:10 p.m.)


Review request for Ambari, Daniel Gergely, Dmytro Sen, Oliver Szabo, and 
Sebastian Toader.


Bugs: AMBARI-17308
https://issues.apache.org/jira/browse/AMBARI-17308


Repository: ambari


Description
---

In logsearch_feeder service log, we got errors like below
```
2016-06-20 15:28:09,368 ERROR file=ambari-audit.log 
org.apache.ambari.logfeeder.mapper.MapperDate LogFeederUtil.java:356 - Error 
applying date transformation. isEpoch=false, 
dateFormat=-MM-dd'T'HH:mm:ss.SSSZ, value=2016-06-20T15:28:08.000. 
mapClass=map_date, input=input:source=file, 
path=/var/log/ambari-server/ambari-audit.log, fieldName=logtime. Messages 
suppressed before: 2
java.text.ParseException: Unparseable date: "2016-06-20T15:28:08.000"
at java.text.DateFormat.parse(DateFormat.java:366)
at 
org.apache.ambari.logfeeder.mapper.MapperDate.apply(MapperDate.java:83)
at org.apache.ambari.logfeeder.filter.Filter.apply(Filter.java:154)
at 
org.apache.ambari.logfeeder.filter.FilterGrok.applyMessage(FilterGrok.java:291)
at 
org.apache.ambari.logfeeder.filter.FilterGrok.flush(FilterGrok.java:320)
at org.apache.ambari.logfeeder.input.Input.flush(Input.java:125)
at 
org.apache.ambari.logfeeder.input.InputFile.processFile(InputFile.java:430)
at org.apache.ambari.logfeeder.input.InputFile.start(InputFile.java:260)
at org.apache.ambari.logfeeder.input.Input.run(Input.java:100)
at java.lang.Thread.run(Thread.java:745)
```
ambari-audit.log is like below
```
2016-07-21T01:52:49.875+09, User(admin), RemoteIp(192.168.72.1), 
Operation(Repository update), RequestType(PUT), 
url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
 ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
id(HDP-2.5), Base 
URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
2016-07-21T01:52:49.905+09, User(admin), RemoteIp(192.168.72.1), 
Operation(Repository update), RequestType(PUT), 
url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu16/repositories/HDP-2.5),
 ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu16), Repo 
id(HDP-2.5), Base 
URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu16/2.x/BUILDS/2.5.0.0-1025)
2016-07-21T01:52:50.015+09, User(admin), RemoteIp(192.168.72.1), 
Operation(Repository update), RequestType(PUT), 
url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-UTILS-1.1.0.21),
 ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
id(HDP-UTILS-1.1.0.21), Base 
URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/ubuntu14)
```
I think date format of the ambari-audit.log (2016-07-21T01:52:49.875+09) should 
be like 2016-07-21T01:52:49.875+0900, since grok-pattern can't handle 
2016-07-21T01:52:49.875+09 format.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerDefaultImpl.java
 ea6524f 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/input.config-ambari.json.j2
 5f751bf 

Diff: https://reviews.apache.org/r/50653/diff/


Testing
---

mvn clean test


Thanks,

Masahiro Tanaka

Re: Review Request 50653: AMBARI-17308 Ambari Logfeeder outputs a lot of errors due to parse date

2016-08-03 Thread Oliver Szabo 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50653/#review144684
---



Thank you for the contribution.

Merged into trunk and branch-2.4:
trunk: bfa24d139aa9d10925850165702024e7a094e473 
branch-2.4: c41565852d93f1af6cf0278d752b839689948f15

please add "trunk, branch-2.4" to the "Branch:" tag in the review request.
Then you can close (submit) this review request.

- Oliver Szabo


On Aug. 3, 2016, 12:36 p.m., Masahiro Tanaka wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50653/
> ---
> 
> (Updated Aug. 3, 2016, 12:36 p.m.)
> 
> 
> Review request for Ambari, Daniel Gergely, Dmytro Sen, Oliver Szabo, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-17308
> https://issues.apache.org/jira/browse/AMBARI-17308
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> In logsearch_feeder service log, we got errors like below
> ```
> 2016-06-20 15:28:09,368 ERROR file=ambari-audit.log 
> org.apache.ambari.logfeeder.mapper.MapperDate LogFeederUtil.java:356 - Error 
> applying date transformation. isEpoch=false, 
> dateFormat=-MM-dd'T'HH:mm:ss.SSSZ, value=2016-06-20T15:28:08.000. 
> mapClass=map_date, input=input:source=file, 
> path=/var/log/ambari-server/ambari-audit.log, fieldName=logtime. Messages 
> suppressed before: 2
> java.text.ParseException: Unparseable date: "2016-06-20T15:28:08.000"
>   at java.text.DateFormat.parse(DateFormat.java:366)
>   at 
> org.apache.ambari.logfeeder.mapper.MapperDate.apply(MapperDate.java:83)
>   at org.apache.ambari.logfeeder.filter.Filter.apply(Filter.java:154)
>   at 
> org.apache.ambari.logfeeder.filter.FilterGrok.applyMessage(FilterGrok.java:291)
>   at 
> org.apache.ambari.logfeeder.filter.FilterGrok.flush(FilterGrok.java:320)
>   at org.apache.ambari.logfeeder.input.Input.flush(Input.java:125)
>   at 
> org.apache.ambari.logfeeder.input.InputFile.processFile(InputFile.java:430)
>   at org.apache.ambari.logfeeder.input.InputFile.start(InputFile.java:260)
>   at org.apache.ambari.logfeeder.input.Input.run(Input.java:100)
>   at java.lang.Thread.run(Thread.java:745)
> ```
> ambari-audit.log is like below
> ```
> 2016-07-21T01:52:49.875+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
> 2016-07-21T01:52:49.905+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu16/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu16), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu16/2.x/BUILDS/2.5.0.0-1025)
> 2016-07-21T01:52:50.015+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-UTILS-1.1.0.21),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-UTILS-1.1.0.21), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/ubuntu14)
> ```
> I think date format of the ambari-audit.log (2016-07-21T01:52:49.875+09) 
> should be like 2016-07-21T01:52:49.875+0900, since grok-pattern can't handle 
> 2016-07-21T01:52:49.875+09 format.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerDefaultImpl.java
>  ea6524f 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/input.config-ambari.json.j2
>  5f751bf 
> 
> Diff: https://reviews.apache.org/r/50653/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Masahiro Tanaka
> 
>

Re: Review Request 50653: AMBARI-17308 Ambari Logfeeder outputs a lot of errors due to parse date

2016-08-03 Thread Masahiro Tanaka 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50653/
---

(Updated Aug. 3, 2016, 9:36 p.m.)


Review request for Ambari, Daniel Gergely, Dmytro Sen, Oliver Szabo, and 
Sebastian Toader.


Changes
---

Updated the patch


Bugs: AMBARI-17308
https://issues.apache.org/jira/browse/AMBARI-17308


Repository: ambari


Description
---

In logsearch_feeder service log, we got errors like below
```
2016-06-20 15:28:09,368 ERROR file=ambari-audit.log 
org.apache.ambari.logfeeder.mapper.MapperDate LogFeederUtil.java:356 - Error 
applying date transformation. isEpoch=false, 
dateFormat=-MM-dd'T'HH:mm:ss.SSSZ, value=2016-06-20T15:28:08.000. 
mapClass=map_date, input=input:source=file, 
path=/var/log/ambari-server/ambari-audit.log, fieldName=logtime. Messages 
suppressed before: 2
java.text.ParseException: Unparseable date: "2016-06-20T15:28:08.000"
at java.text.DateFormat.parse(DateFormat.java:366)
at 
org.apache.ambari.logfeeder.mapper.MapperDate.apply(MapperDate.java:83)
at org.apache.ambari.logfeeder.filter.Filter.apply(Filter.java:154)
at 
org.apache.ambari.logfeeder.filter.FilterGrok.applyMessage(FilterGrok.java:291)
at 
org.apache.ambari.logfeeder.filter.FilterGrok.flush(FilterGrok.java:320)
at org.apache.ambari.logfeeder.input.Input.flush(Input.java:125)
at 
org.apache.ambari.logfeeder.input.InputFile.processFile(InputFile.java:430)
at org.apache.ambari.logfeeder.input.InputFile.start(InputFile.java:260)
at org.apache.ambari.logfeeder.input.Input.run(Input.java:100)
at java.lang.Thread.run(Thread.java:745)
```
ambari-audit.log is like below
```
2016-07-21T01:52:49.875+09, User(admin), RemoteIp(192.168.72.1), 
Operation(Repository update), RequestType(PUT), 
url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
 ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
id(HDP-2.5), Base 
URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
2016-07-21T01:52:49.905+09, User(admin), RemoteIp(192.168.72.1), 
Operation(Repository update), RequestType(PUT), 
url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu16/repositories/HDP-2.5),
 ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu16), Repo 
id(HDP-2.5), Base 
URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu16/2.x/BUILDS/2.5.0.0-1025)
2016-07-21T01:52:50.015+09, User(admin), RemoteIp(192.168.72.1), 
Operation(Repository update), RequestType(PUT), 
url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-UTILS-1.1.0.21),
 ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
id(HDP-UTILS-1.1.0.21), Base 
URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/ubuntu14)
```
I think date format of the ambari-audit.log (2016-07-21T01:52:49.875+09) should 
be like 2016-07-21T01:52:49.875+0900, since grok-pattern can't handle 
2016-07-21T01:52:49.875+09 format.


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerDefaultImpl.java
 ea6524f 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/input.config-ambari.json.j2
 5f751bf 

Diff: https://reviews.apache.org/r/50653/diff/


Testing
---

mvn clean test


Thanks,

Masahiro Tanaka

Re: Review Request 50653: AMBARI-17308 Ambari Logfeeder outputs a lot of errors due to parse date

2016-08-03 Thread Oliver Szabo 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50653/#review144594
---


Fix it, then Ship it!





ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/input.config-ambari.json.j2
 (line 245)


I think we can use XX here too, just keep it consistent


- Oliver Szabo


On Aug. 3, 2016, 1:15 a.m., Masahiro Tanaka wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50653/
> ---
> 
> (Updated Aug. 3, 2016, 1:15 a.m.)
> 
> 
> Review request for Ambari, Daniel Gergely, Dmytro Sen, Oliver Szabo, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-17308
> https://issues.apache.org/jira/browse/AMBARI-17308
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> In logsearch_feeder service log, we got errors like below
> ```
> 2016-06-20 15:28:09,368 ERROR file=ambari-audit.log 
> org.apache.ambari.logfeeder.mapper.MapperDate LogFeederUtil.java:356 - Error 
> applying date transformation. isEpoch=false, 
> dateFormat=-MM-dd'T'HH:mm:ss.SSSZ, value=2016-06-20T15:28:08.000. 
> mapClass=map_date, input=input:source=file, 
> path=/var/log/ambari-server/ambari-audit.log, fieldName=logtime. Messages 
> suppressed before: 2
> java.text.ParseException: Unparseable date: "2016-06-20T15:28:08.000"
>   at java.text.DateFormat.parse(DateFormat.java:366)
>   at 
> org.apache.ambari.logfeeder.mapper.MapperDate.apply(MapperDate.java:83)
>   at org.apache.ambari.logfeeder.filter.Filter.apply(Filter.java:154)
>   at 
> org.apache.ambari.logfeeder.filter.FilterGrok.applyMessage(FilterGrok.java:291)
>   at 
> org.apache.ambari.logfeeder.filter.FilterGrok.flush(FilterGrok.java:320)
>   at org.apache.ambari.logfeeder.input.Input.flush(Input.java:125)
>   at 
> org.apache.ambari.logfeeder.input.InputFile.processFile(InputFile.java:430)
>   at org.apache.ambari.logfeeder.input.InputFile.start(InputFile.java:260)
>   at org.apache.ambari.logfeeder.input.Input.run(Input.java:100)
>   at java.lang.Thread.run(Thread.java:745)
> ```
> ambari-audit.log is like below
> ```
> 2016-07-21T01:52:49.875+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
> 2016-07-21T01:52:49.905+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu16/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu16), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu16/2.x/BUILDS/2.5.0.0-1025)
> 2016-07-21T01:52:50.015+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-UTILS-1.1.0.21),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-UTILS-1.1.0.21), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/ubuntu14)
> ```
> I think date format of the ambari-audit.log (2016-07-21T01:52:49.875+09) 
> should be like 2016-07-21T01:52:49.875+0900, since grok-pattern can't handle 
> 2016-07-21T01:52:49.875+09 format.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerDefaultImpl.java
>  ea6524f 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/input.config-ambari.json.j2
>  5f751bf 
> 
> Diff: https://reviews.apache.org/r/50653/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Masahiro Tanaka
> 
>

Re: Review Request 50653: AMBARI-17308 Ambari Logfeeder outputs a lot of errors due to parse date

2016-08-03 Thread Sebastian Toader 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50653/#review144587
---


Ship it!




Ship It!

- Sebastian Toader


On Aug. 3, 2016, 3:15 a.m., Masahiro Tanaka wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50653/
> ---
> 
> (Updated Aug. 3, 2016, 3:15 a.m.)
> 
> 
> Review request for Ambari, Daniel Gergely, Dmytro Sen, Oliver Szabo, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-17308
> https://issues.apache.org/jira/browse/AMBARI-17308
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> In logsearch_feeder service log, we got errors like below
> ```
> 2016-06-20 15:28:09,368 ERROR file=ambari-audit.log 
> org.apache.ambari.logfeeder.mapper.MapperDate LogFeederUtil.java:356 - Error 
> applying date transformation. isEpoch=false, 
> dateFormat=-MM-dd'T'HH:mm:ss.SSSZ, value=2016-06-20T15:28:08.000. 
> mapClass=map_date, input=input:source=file, 
> path=/var/log/ambari-server/ambari-audit.log, fieldName=logtime. Messages 
> suppressed before: 2
> java.text.ParseException: Unparseable date: "2016-06-20T15:28:08.000"
>   at java.text.DateFormat.parse(DateFormat.java:366)
>   at 
> org.apache.ambari.logfeeder.mapper.MapperDate.apply(MapperDate.java:83)
>   at org.apache.ambari.logfeeder.filter.Filter.apply(Filter.java:154)
>   at 
> org.apache.ambari.logfeeder.filter.FilterGrok.applyMessage(FilterGrok.java:291)
>   at 
> org.apache.ambari.logfeeder.filter.FilterGrok.flush(FilterGrok.java:320)
>   at org.apache.ambari.logfeeder.input.Input.flush(Input.java:125)
>   at 
> org.apache.ambari.logfeeder.input.InputFile.processFile(InputFile.java:430)
>   at org.apache.ambari.logfeeder.input.InputFile.start(InputFile.java:260)
>   at org.apache.ambari.logfeeder.input.Input.run(Input.java:100)
>   at java.lang.Thread.run(Thread.java:745)
> ```
> ambari-audit.log is like below
> ```
> 2016-07-21T01:52:49.875+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
> 2016-07-21T01:52:49.905+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu16/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu16), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu16/2.x/BUILDS/2.5.0.0-1025)
> 2016-07-21T01:52:50.015+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-UTILS-1.1.0.21),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-UTILS-1.1.0.21), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/ubuntu14)
> ```
> I think date format of the ambari-audit.log (2016-07-21T01:52:49.875+09) 
> should be like 2016-07-21T01:52:49.875+0900, since grok-pattern can't handle 
> 2016-07-21T01:52:49.875+09 format.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerDefaultImpl.java
>  ea6524f 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/input.config-ambari.json.j2
>  5f751bf 
> 
> Diff: https://reviews.apache.org/r/50653/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Masahiro Tanaka
> 
>

Re: Review Request 50653: AMBARI-17308 Ambari Logfeeder outputs a lot of errors due to parse date

2016-08-02 Thread Masahiro Tanaka 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50653/
---

(Updated Aug. 3, 2016, 10:15 a.m.)


Review request for Ambari, Daniel Gergely, Dmytro Sen, Oliver Szabo, and 
Sebastian Toader.


Changes
---

Update the patch


Bugs: AMBARI-17308
https://issues.apache.org/jira/browse/AMBARI-17308


Repository: ambari


Description
---

In logsearch_feeder service log, we got errors like below
```
2016-06-20 15:28:09,368 ERROR file=ambari-audit.log 
org.apache.ambari.logfeeder.mapper.MapperDate LogFeederUtil.java:356 - Error 
applying date transformation. isEpoch=false, 
dateFormat=-MM-dd'T'HH:mm:ss.SSSZ, value=2016-06-20T15:28:08.000. 
mapClass=map_date, input=input:source=file, 
path=/var/log/ambari-server/ambari-audit.log, fieldName=logtime. Messages 
suppressed before: 2
java.text.ParseException: Unparseable date: "2016-06-20T15:28:08.000"
at java.text.DateFormat.parse(DateFormat.java:366)
at 
org.apache.ambari.logfeeder.mapper.MapperDate.apply(MapperDate.java:83)
at org.apache.ambari.logfeeder.filter.Filter.apply(Filter.java:154)
at 
org.apache.ambari.logfeeder.filter.FilterGrok.applyMessage(FilterGrok.java:291)
at 
org.apache.ambari.logfeeder.filter.FilterGrok.flush(FilterGrok.java:320)
at org.apache.ambari.logfeeder.input.Input.flush(Input.java:125)
at 
org.apache.ambari.logfeeder.input.InputFile.processFile(InputFile.java:430)
at org.apache.ambari.logfeeder.input.InputFile.start(InputFile.java:260)
at org.apache.ambari.logfeeder.input.Input.run(Input.java:100)
at java.lang.Thread.run(Thread.java:745)
```
ambari-audit.log is like below
```
2016-07-21T01:52:49.875+09, User(admin), RemoteIp(192.168.72.1), 
Operation(Repository update), RequestType(PUT), 
url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
 ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
id(HDP-2.5), Base 
URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
2016-07-21T01:52:49.905+09, User(admin), RemoteIp(192.168.72.1), 
Operation(Repository update), RequestType(PUT), 
url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu16/repositories/HDP-2.5),
 ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu16), Repo 
id(HDP-2.5), Base 
URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu16/2.x/BUILDS/2.5.0.0-1025)
2016-07-21T01:52:50.015+09, User(admin), RemoteIp(192.168.72.1), 
Operation(Repository update), RequestType(PUT), 
url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-UTILS-1.1.0.21),
 ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
id(HDP-UTILS-1.1.0.21), Base 
URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/ubuntu14)
```
I think date format of the ambari-audit.log (2016-07-21T01:52:49.875+09) should 
be like 2016-07-21T01:52:49.875+0900, since grok-pattern can't handle 
2016-07-21T01:52:49.875+09 format.


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerDefaultImpl.java
 ea6524f 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/input.config-ambari.json.j2
 5f751bf 

Diff: https://reviews.apache.org/r/50653/diff/


Testing
---

mvn clean test


Thanks,

Masahiro Tanaka

Re: Review Request 50653: AMBARI-17308 Ambari Logfeeder outputs a lot of errors due to parse date

2016-08-02 Thread Masahiro Tanaka 


> On 8月 2, 2016, 1:58 p.m., Sebastian Toader wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerDefaultImpl.java,
> >  line 51
> > 
> >
> > The 'X' stands for the ISO timezone representation. 
> > 
> > I think grok supports ISO timezone format as well. Can you confirm that 
> > grok patterns do not support iso 8061 timezone format?
> > 
> > If it doesn't than go ahead and change for X to Z
> 
> Masahiro Tanaka wrote:
> Thank you for reviewing! I used [this 
> site](https://grokdebug.herokuapp.com/) to check if the pattern is able to 
> parse the log.
> Please try pattern A and pattern B below.
> 
> Pattern A:
> ```
> # input
> 2016-07-21T01:52:49.875+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
> 
> # pattern
> (?m)^%{TIMESTAMP_ISO8601:logtime},%{SPACE}%{GREEDYDATA:log_message}
> ```
> 
> Pattern B:
> ```
> # input (timestamp is different from above, others are the same)
> 2016-07-21T01:52:49.875+0900, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
> 
> # pattern
> (?m)^%{TIMESTAMP_ISO8601:logtime},%{SPACE}%{GREEDYDATA:log_message}
> ```
> 
> Sebastian Toader wrote:
> Can you try ```return new 
> SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSS**XX**");```? According to the 
> documentation is a single pattern letter is used than any fraction of an hour 
> is ignored. For example, if the pattern is "X" and the time zone is 
> "GMT+05:30", "+05" is produced.
> 
> Sebastian Toader wrote:
> I meant: SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSSXX")
> 
> Masahiro Tanaka wrote:
> I've tried that. The result (`/var/log/ambari-server/ambari-audit.log` is 
> like as below
> ```
> 2016-08-02T21:57:58.799+0900, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/redhat6/repositories/HDP-UTILS-1.1.0.21),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(redhat6), Repo 
> id(HDP-UTILS-1.1.0.21), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos6)
> 2016-08-02T21:57:58.810+0900, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/redhat7/repositories/HDP-UTILS-1.1.0.21),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(redhat7), Repo 
> id(HDP-UTILS-1.1.0.21), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7)
> 2016-08-02T21:57:58.816+0900, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/debian7/repositories/HDP-UTILS-1.1.0.21),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(debian7), Repo 
> id(HDP-UTILS-1.1.0.21), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/debian7)
> 2016-08-02T21:57:58.823+0900, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/redhat6/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(redhat6), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/centos6/2.x/BUILDS/2.5.0.0-1116)
> ```
> 
> I think this format is same as 
> `SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSSZ");` at least in JST.
> 
> Oliver Szabo wrote:
> in SimpleDateFormat:
> Z Time zone   RFC 822 time zone   -0800
> X Time zone   ISO 8601 time zone  -08; -0800; -08:00
> 
> It would make sense to use XX instead of Z, because in the grok pattern 
> we uses ISO 8601 format for parsing (to keep it clear what is the format here)

OK, I'll update the patch to use "XX".


- Masahiro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50653/#review144381
---

Re: Review Request 50653: AMBARI-17308 Ambari Logfeeder outputs a lot of errors due to parse date

2016-08-02 Thread Oliver Szabo 


> On Aug. 2, 2016, 4:58 a.m., Sebastian Toader wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerDefaultImpl.java,
> >  line 51
> > 
> >
> > The 'X' stands for the ISO timezone representation. 
> > 
> > I think grok supports ISO timezone format as well. Can you confirm that 
> > grok patterns do not support iso 8061 timezone format?
> > 
> > If it doesn't than go ahead and change for X to Z
> 
> Masahiro Tanaka wrote:
> Thank you for reviewing! I used [this 
> site](https://grokdebug.herokuapp.com/) to check if the pattern is able to 
> parse the log.
> Please try pattern A and pattern B below.
> 
> Pattern A:
> ```
> # input
> 2016-07-21T01:52:49.875+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
> 
> # pattern
> (?m)^%{TIMESTAMP_ISO8601:logtime},%{SPACE}%{GREEDYDATA:log_message}
> ```
> 
> Pattern B:
> ```
> # input (timestamp is different from above, others are the same)
> 2016-07-21T01:52:49.875+0900, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
> 
> # pattern
> (?m)^%{TIMESTAMP_ISO8601:logtime},%{SPACE}%{GREEDYDATA:log_message}
> ```
> 
> Sebastian Toader wrote:
> Can you try ```return new 
> SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSS**XX**");```? According to the 
> documentation is a single pattern letter is used than any fraction of an hour 
> is ignored. For example, if the pattern is "X" and the time zone is 
> "GMT+05:30", "+05" is produced.
> 
> Sebastian Toader wrote:
> I meant: SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSSXX")
> 
> Masahiro Tanaka wrote:
> I've tried that. The result (`/var/log/ambari-server/ambari-audit.log` is 
> like as below
> ```
> 2016-08-02T21:57:58.799+0900, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/redhat6/repositories/HDP-UTILS-1.1.0.21),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(redhat6), Repo 
> id(HDP-UTILS-1.1.0.21), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos6)
> 2016-08-02T21:57:58.810+0900, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/redhat7/repositories/HDP-UTILS-1.1.0.21),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(redhat7), Repo 
> id(HDP-UTILS-1.1.0.21), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7)
> 2016-08-02T21:57:58.816+0900, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/debian7/repositories/HDP-UTILS-1.1.0.21),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(debian7), Repo 
> id(HDP-UTILS-1.1.0.21), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/debian7)
> 2016-08-02T21:57:58.823+0900, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/redhat6/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(redhat6), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/centos6/2.x/BUILDS/2.5.0.0-1116)
> ```
> 
> I think this format is same as 
> `SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSSZ");` at least in JST.

in SimpleDateFormat:
Z   Time zone   RFC 822 time zone   -0800
X   Time zone   ISO 8601 time zone  -08; -0800; -08:00

It would make sense to use XX instead of Z, because in the grok pattern we uses 
ISO 8601 format for parsing (to keep it clear what is the format here)


- Oliver


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50653/#review144381
---


On Aug. 1, 2016, 10:23 a.m., Masahiro Tanaka wrote:
> 
>

Re: Review Request 50653: AMBARI-17308 Ambari Logfeeder outputs a lot of errors due to parse date

2016-08-02 Thread Masahiro Tanaka 


> On Aug. 2, 2016, 1:58 p.m., Sebastian Toader wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerDefaultImpl.java,
> >  line 51
> > 
> >
> > The 'X' stands for the ISO timezone representation. 
> > 
> > I think grok supports ISO timezone format as well. Can you confirm that 
> > grok patterns do not support iso 8061 timezone format?
> > 
> > If it doesn't than go ahead and change for X to Z
> 
> Masahiro Tanaka wrote:
> Thank you for reviewing! I used [this 
> site](https://grokdebug.herokuapp.com/) to check if the pattern is able to 
> parse the log.
> Please try pattern A and pattern B below.
> 
> Pattern A:
> ```
> # input
> 2016-07-21T01:52:49.875+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
> 
> # pattern
> (?m)^%{TIMESTAMP_ISO8601:logtime},%{SPACE}%{GREEDYDATA:log_message}
> ```
> 
> Pattern B:
> ```
> # input (timestamp is different from above, others are the same)
> 2016-07-21T01:52:49.875+0900, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
> 
> # pattern
> (?m)^%{TIMESTAMP_ISO8601:logtime},%{SPACE}%{GREEDYDATA:log_message}
> ```
> 
> Sebastian Toader wrote:
> Can you try ```return new 
> SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSS**XX**");```? According to the 
> documentation is a single pattern letter is used than any fraction of an hour 
> is ignored. For example, if the pattern is "X" and the time zone is 
> "GMT+05:30", "+05" is produced.
> 
> Sebastian Toader wrote:
> I meant: SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSSXX")

I've tried that. The result (`/var/log/ambari-server/ambari-audit.log` is like 
as below
```
2016-08-02T21:57:58.799+0900, User(admin), RemoteIp(192.168.72.1), 
Operation(Repository update), RequestType(PUT), 
url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/redhat6/repositories/HDP-UTILS-1.1.0.21),
 ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(redhat6), Repo 
id(HDP-UTILS-1.1.0.21), Base 
URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos6)
2016-08-02T21:57:58.810+0900, User(admin), RemoteIp(192.168.72.1), 
Operation(Repository update), RequestType(PUT), 
url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/redhat7/repositories/HDP-UTILS-1.1.0.21),
 ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(redhat7), Repo 
id(HDP-UTILS-1.1.0.21), Base 
URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7)
2016-08-02T21:57:58.816+0900, User(admin), RemoteIp(192.168.72.1), 
Operation(Repository update), RequestType(PUT), 
url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/debian7/repositories/HDP-UTILS-1.1.0.21),
 ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(debian7), Repo 
id(HDP-UTILS-1.1.0.21), Base 
URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/debian7)
2016-08-02T21:57:58.823+0900, User(admin), RemoteIp(192.168.72.1), 
Operation(Repository update), RequestType(PUT), 
url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/redhat6/repositories/HDP-2.5),
 ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(redhat6), Repo 
id(HDP-2.5), Base 
URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/centos6/2.x/BUILDS/2.5.0.0-1116)
```

I think this format is same as 
`SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSSZ");` at least in JST.


- Masahiro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50653/#review144381
---


On Aug. 1, 2016, 7:23 p.m., Masahiro Tanaka wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50653/
> ---
> 
> (Updated Aug. 1, 2016, 7:23 p.m.)
> 
> 
> Review request for Ambari, Daniel Gergely, Dmytro Sen, Oliver Szabo, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-17308
> https://issues.apache.org/jira/browse/AMBARI-17308

Re: Review Request 50653: AMBARI-17308 Ambari Logfeeder outputs a lot of errors due to parse date

2016-08-02 Thread Sebastian Toader 


> On Aug. 2, 2016, 6:58 a.m., Sebastian Toader wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerDefaultImpl.java,
> >  line 51
> > 
> >
> > The 'X' stands for the ISO timezone representation. 
> > 
> > I think grok supports ISO timezone format as well. Can you confirm that 
> > grok patterns do not support iso 8061 timezone format?
> > 
> > If it doesn't than go ahead and change for X to Z
> 
> Masahiro Tanaka wrote:
> Thank you for reviewing! I used [this 
> site](https://grokdebug.herokuapp.com/) to check if the pattern is able to 
> parse the log.
> Please try pattern A and pattern B below.
> 
> Pattern A:
> ```
> # input
> 2016-07-21T01:52:49.875+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
> 
> # pattern
> (?m)^%{TIMESTAMP_ISO8601:logtime},%{SPACE}%{GREEDYDATA:log_message}
> ```
> 
> Pattern B:
> ```
> # input (timestamp is different from above, others are the same)
> 2016-07-21T01:52:49.875+0900, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
> 
> # pattern
> (?m)^%{TIMESTAMP_ISO8601:logtime},%{SPACE}%{GREEDYDATA:log_message}
> ```
> 
> Sebastian Toader wrote:
> Can you try ```return new 
> SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSS**XX**");```? According to the 
> documentation is a single pattern letter is used than any fraction of an hour 
> is ignored. For example, if the pattern is "X" and the time zone is 
> "GMT+05:30", "+05" is produced.

I meant: SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSSXX")


- Sebastian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50653/#review144381
---


On Aug. 1, 2016, 12:23 p.m., Masahiro Tanaka wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50653/
> ---
> 
> (Updated Aug. 1, 2016, 12:23 p.m.)
> 
> 
> Review request for Ambari, Daniel Gergely, Dmytro Sen, Oliver Szabo, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-17308
> https://issues.apache.org/jira/browse/AMBARI-17308
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> In logsearch_feeder service log, we got errors like below
> ```
> 2016-06-20 15:28:09,368 ERROR file=ambari-audit.log 
> org.apache.ambari.logfeeder.mapper.MapperDate LogFeederUtil.java:356 - Error 
> applying date transformation. isEpoch=false, 
> dateFormat=-MM-dd'T'HH:mm:ss.SSSZ, value=2016-06-20T15:28:08.000. 
> mapClass=map_date, input=input:source=file, 
> path=/var/log/ambari-server/ambari-audit.log, fieldName=logtime. Messages 
> suppressed before: 2
> java.text.ParseException: Unparseable date: "2016-06-20T15:28:08.000"
>   at java.text.DateFormat.parse(DateFormat.java:366)
>   at 
> org.apache.ambari.logfeeder.mapper.MapperDate.apply(MapperDate.java:83)
>   at org.apache.ambari.logfeeder.filter.Filter.apply(Filter.java:154)
>   at 
> org.apache.ambari.logfeeder.filter.FilterGrok.applyMessage(FilterGrok.java:291)
>   at 
> org.apache.ambari.logfeeder.filter.FilterGrok.flush(FilterGrok.java:320)
>   at org.apache.ambari.logfeeder.input.Input.flush(Input.java:125)
>   at 
> org.apache.ambari.logfeeder.input.InputFile.processFile(InputFile.java:430)
>   at org.apache.ambari.logfeeder.input.InputFile.start(InputFile.java:260)
>   at org.apache.ambari.logfeeder.input.Input.run(Input.java:100)
>   at java.lang.Thread.run(Thread.java:745)
> ```
> ambari-audit.log is like below
> ```
> 2016-07-21T01:52:49.875+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
> 2016-07-21T01:52:49.905+09, User(admin), RemoteIp(192.168.72.1), 
>

Re: Review Request 50653: AMBARI-17308 Ambari Logfeeder outputs a lot of errors due to parse date

2016-08-02 Thread Sebastian Toader 


> On Aug. 2, 2016, 6:58 a.m., Sebastian Toader wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerDefaultImpl.java,
> >  line 51
> > 
> >
> > The 'X' stands for the ISO timezone representation. 
> > 
> > I think grok supports ISO timezone format as well. Can you confirm that 
> > grok patterns do not support iso 8061 timezone format?
> > 
> > If it doesn't than go ahead and change for X to Z
> 
> Masahiro Tanaka wrote:
> Thank you for reviewing! I used [this 
> site](https://grokdebug.herokuapp.com/) to check if the pattern is able to 
> parse the log.
> Please try pattern A and pattern B below.
> 
> Pattern A:
> ```
> # input
> 2016-07-21T01:52:49.875+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
> 
> # pattern
> (?m)^%{TIMESTAMP_ISO8601:logtime},%{SPACE}%{GREEDYDATA:log_message}
> ```
> 
> Pattern B:
> ```
> # input (timestamp is different from above, others are the same)
> 2016-07-21T01:52:49.875+0900, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
> 
> # pattern
> (?m)^%{TIMESTAMP_ISO8601:logtime},%{SPACE}%{GREEDYDATA:log_message}
> ```

Can you try ```return new 
SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSS**XX**");```? According to the 
documentation is a single pattern letter is used than any fraction of an hour 
is ignored. For example, if the pattern is "X" and the time zone is 
"GMT+05:30", "+05" is produced.


- Sebastian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50653/#review144381
---


On Aug. 1, 2016, 12:23 p.m., Masahiro Tanaka wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50653/
> ---
> 
> (Updated Aug. 1, 2016, 12:23 p.m.)
> 
> 
> Review request for Ambari, Daniel Gergely, Dmytro Sen, Oliver Szabo, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-17308
> https://issues.apache.org/jira/browse/AMBARI-17308
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> In logsearch_feeder service log, we got errors like below
> ```
> 2016-06-20 15:28:09,368 ERROR file=ambari-audit.log 
> org.apache.ambari.logfeeder.mapper.MapperDate LogFeederUtil.java:356 - Error 
> applying date transformation. isEpoch=false, 
> dateFormat=-MM-dd'T'HH:mm:ss.SSSZ, value=2016-06-20T15:28:08.000. 
> mapClass=map_date, input=input:source=file, 
> path=/var/log/ambari-server/ambari-audit.log, fieldName=logtime. Messages 
> suppressed before: 2
> java.text.ParseException: Unparseable date: "2016-06-20T15:28:08.000"
>   at java.text.DateFormat.parse(DateFormat.java:366)
>   at 
> org.apache.ambari.logfeeder.mapper.MapperDate.apply(MapperDate.java:83)
>   at org.apache.ambari.logfeeder.filter.Filter.apply(Filter.java:154)
>   at 
> org.apache.ambari.logfeeder.filter.FilterGrok.applyMessage(FilterGrok.java:291)
>   at 
> org.apache.ambari.logfeeder.filter.FilterGrok.flush(FilterGrok.java:320)
>   at org.apache.ambari.logfeeder.input.Input.flush(Input.java:125)
>   at 
> org.apache.ambari.logfeeder.input.InputFile.processFile(InputFile.java:430)
>   at org.apache.ambari.logfeeder.input.InputFile.start(InputFile.java:260)
>   at org.apache.ambari.logfeeder.input.Input.run(Input.java:100)
>   at java.lang.Thread.run(Thread.java:745)
> ```
> ambari-audit.log is like below
> ```
> 2016-07-21T01:52:49.875+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
> 2016-07-21T01:52:49.905+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
>

Re: Review Request 50653: AMBARI-17308 Ambari Logfeeder outputs a lot of errors due to parse date

2016-08-02 Thread Masahiro Tanaka 


> On Aug. 2, 2016, 1:58 p.m., Sebastian Toader wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerDefaultImpl.java,
> >  line 51
> > 
> >
> > The 'X' stands for the ISO timezone representation. 
> > 
> > I think grok supports ISO timezone format as well. Can you confirm that 
> > grok patterns do not support iso 8061 timezone format?
> > 
> > If it doesn't than go ahead and change for X to Z

Thank you for reviewing! I used [this site](https://grokdebug.herokuapp.com/) 
to check if the pattern is able to parse the log.
Please try pattern A and pattern B below.

Pattern A:
```
# input
2016-07-21T01:52:49.875+09, User(admin), RemoteIp(192.168.72.1), 
Operation(Repository update), RequestType(PUT), 
url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
 ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
id(HDP-2.5), Base 
URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)

# pattern
(?m)^%{TIMESTAMP_ISO8601:logtime},%{SPACE}%{GREEDYDATA:log_message}
```

Pattern B:
```
# input (timestamp is different from above, others are the same)
2016-07-21T01:52:49.875+0900, User(admin), RemoteIp(192.168.72.1), 
Operation(Repository update), RequestType(PUT), 
url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
 ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
id(HDP-2.5), Base 
URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)

# pattern
(?m)^%{TIMESTAMP_ISO8601:logtime},%{SPACE}%{GREEDYDATA:log_message}
```


- Masahiro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50653/#review144381
---


On Aug. 1, 2016, 7:23 p.m., Masahiro Tanaka wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50653/
> ---
> 
> (Updated Aug. 1, 2016, 7:23 p.m.)
> 
> 
> Review request for Ambari, Daniel Gergely, Dmytro Sen, Oliver Szabo, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-17308
> https://issues.apache.org/jira/browse/AMBARI-17308
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> In logsearch_feeder service log, we got errors like below
> ```
> 2016-06-20 15:28:09,368 ERROR file=ambari-audit.log 
> org.apache.ambari.logfeeder.mapper.MapperDate LogFeederUtil.java:356 - Error 
> applying date transformation. isEpoch=false, 
> dateFormat=-MM-dd'T'HH:mm:ss.SSSZ, value=2016-06-20T15:28:08.000. 
> mapClass=map_date, input=input:source=file, 
> path=/var/log/ambari-server/ambari-audit.log, fieldName=logtime. Messages 
> suppressed before: 2
> java.text.ParseException: Unparseable date: "2016-06-20T15:28:08.000"
>   at java.text.DateFormat.parse(DateFormat.java:366)
>   at 
> org.apache.ambari.logfeeder.mapper.MapperDate.apply(MapperDate.java:83)
>   at org.apache.ambari.logfeeder.filter.Filter.apply(Filter.java:154)
>   at 
> org.apache.ambari.logfeeder.filter.FilterGrok.applyMessage(FilterGrok.java:291)
>   at 
> org.apache.ambari.logfeeder.filter.FilterGrok.flush(FilterGrok.java:320)
>   at org.apache.ambari.logfeeder.input.Input.flush(Input.java:125)
>   at 
> org.apache.ambari.logfeeder.input.InputFile.processFile(InputFile.java:430)
>   at org.apache.ambari.logfeeder.input.InputFile.start(InputFile.java:260)
>   at org.apache.ambari.logfeeder.input.Input.run(Input.java:100)
>   at java.lang.Thread.run(Thread.java:745)
> ```
> ambari-audit.log is like below
> ```
> 2016-07-21T01:52:49.875+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
> 2016-07-21T01:52:49.905+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu16/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu16), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu16/2.x/BUILDS/2.5.0.0-1025)
> 2016-07-21T01:52:50.015+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
>

Re: Review Request 50653: AMBARI-17308 Ambari Logfeeder outputs a lot of errors due to parse date

2016-08-01 Thread Sebastian Toader 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50653/#review144381
---




ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerDefaultImpl.java
 (line 51)


The 'X' stands for the ISO timezone representation. 

I think grok supports ISO timezone format as well. Can you confirm that 
grok patterns do not support iso 8061 timezone format?

If it doesn't than go ahead and change for X to Z


- Sebastian Toader


On Aug. 1, 2016, 12:23 p.m., Masahiro Tanaka wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50653/
> ---
> 
> (Updated Aug. 1, 2016, 12:23 p.m.)
> 
> 
> Review request for Ambari, Daniel Gergely, Dmytro Sen, Oliver Szabo, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-17308
> https://issues.apache.org/jira/browse/AMBARI-17308
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> In logsearch_feeder service log, we got errors like below
> ```
> 2016-06-20 15:28:09,368 ERROR file=ambari-audit.log 
> org.apache.ambari.logfeeder.mapper.MapperDate LogFeederUtil.java:356 - Error 
> applying date transformation. isEpoch=false, 
> dateFormat=-MM-dd'T'HH:mm:ss.SSSZ, value=2016-06-20T15:28:08.000. 
> mapClass=map_date, input=input:source=file, 
> path=/var/log/ambari-server/ambari-audit.log, fieldName=logtime. Messages 
> suppressed before: 2
> java.text.ParseException: Unparseable date: "2016-06-20T15:28:08.000"
>   at java.text.DateFormat.parse(DateFormat.java:366)
>   at 
> org.apache.ambari.logfeeder.mapper.MapperDate.apply(MapperDate.java:83)
>   at org.apache.ambari.logfeeder.filter.Filter.apply(Filter.java:154)
>   at 
> org.apache.ambari.logfeeder.filter.FilterGrok.applyMessage(FilterGrok.java:291)
>   at 
> org.apache.ambari.logfeeder.filter.FilterGrok.flush(FilterGrok.java:320)
>   at org.apache.ambari.logfeeder.input.Input.flush(Input.java:125)
>   at 
> org.apache.ambari.logfeeder.input.InputFile.processFile(InputFile.java:430)
>   at org.apache.ambari.logfeeder.input.InputFile.start(InputFile.java:260)
>   at org.apache.ambari.logfeeder.input.Input.run(Input.java:100)
>   at java.lang.Thread.run(Thread.java:745)
> ```
> ambari-audit.log is like below
> ```
> 2016-07-21T01:52:49.875+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
> 2016-07-21T01:52:49.905+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu16/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu16), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu16/2.x/BUILDS/2.5.0.0-1025)
> 2016-07-21T01:52:50.015+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-UTILS-1.1.0.21),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-UTILS-1.1.0.21), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/ubuntu14)
> ```
> I think date format of the ambari-audit.log (2016-07-21T01:52:49.875+09) 
> should be like 2016-07-21T01:52:49.875+0900, since grok-pattern can't handle 
> 2016-07-21T01:52:49.875+09 format.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerDefaultImpl.java
>  ea6524f 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/input.config-ambari.json.j2
>  5f751bf 
> 
> Diff: https://reviews.apache.org/r/50653/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Masahiro Tanaka
> 
>

Re: Review Request 50653: AMBARI-17308 Ambari Logfeeder outputs a lot of errors due to parse date

2016-08-01 Thread Oliver Szabo 


> On Aug. 1, 2016, 12:35 p.m., Oliver Szabo wrote:
> > Ship It!

i will test this out a bit more, my logs looks like this locally:

2016-08-01T17:41:29.739Z, User(null), RemoteIp(192.168.64.1), Operation(User 
login), Roles(
), Status(Failed), Reason(Authentication required) ... based on that the actual 
solution is working, so I check it could work in both cases


- Oliver


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50653/#review144326
---


On Aug. 1, 2016, 10:23 a.m., Masahiro Tanaka wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50653/
> ---
> 
> (Updated Aug. 1, 2016, 10:23 a.m.)
> 
> 
> Review request for Ambari, Daniel Gergely, Dmytro Sen, Oliver Szabo, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-17308
> https://issues.apache.org/jira/browse/AMBARI-17308
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> In logsearch_feeder service log, we got errors like below
> ```
> 2016-06-20 15:28:09,368 ERROR file=ambari-audit.log 
> org.apache.ambari.logfeeder.mapper.MapperDate LogFeederUtil.java:356 - Error 
> applying date transformation. isEpoch=false, 
> dateFormat=-MM-dd'T'HH:mm:ss.SSSZ, value=2016-06-20T15:28:08.000. 
> mapClass=map_date, input=input:source=file, 
> path=/var/log/ambari-server/ambari-audit.log, fieldName=logtime. Messages 
> suppressed before: 2
> java.text.ParseException: Unparseable date: "2016-06-20T15:28:08.000"
>   at java.text.DateFormat.parse(DateFormat.java:366)
>   at 
> org.apache.ambari.logfeeder.mapper.MapperDate.apply(MapperDate.java:83)
>   at org.apache.ambari.logfeeder.filter.Filter.apply(Filter.java:154)
>   at 
> org.apache.ambari.logfeeder.filter.FilterGrok.applyMessage(FilterGrok.java:291)
>   at 
> org.apache.ambari.logfeeder.filter.FilterGrok.flush(FilterGrok.java:320)
>   at org.apache.ambari.logfeeder.input.Input.flush(Input.java:125)
>   at 
> org.apache.ambari.logfeeder.input.InputFile.processFile(InputFile.java:430)
>   at org.apache.ambari.logfeeder.input.InputFile.start(InputFile.java:260)
>   at org.apache.ambari.logfeeder.input.Input.run(Input.java:100)
>   at java.lang.Thread.run(Thread.java:745)
> ```
> ambari-audit.log is like below
> ```
> 2016-07-21T01:52:49.875+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
> 2016-07-21T01:52:49.905+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu16/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu16), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu16/2.x/BUILDS/2.5.0.0-1025)
> 2016-07-21T01:52:50.015+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-UTILS-1.1.0.21),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-UTILS-1.1.0.21), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/ubuntu14)
> ```
> I think date format of the ambari-audit.log (2016-07-21T01:52:49.875+09) 
> should be like 2016-07-21T01:52:49.875+0900, since grok-pattern can't handle 
> 2016-07-21T01:52:49.875+09 format.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerDefaultImpl.java
>  ea6524f 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/input.config-ambari.json.j2
>  5f751bf 
> 
> Diff: https://reviews.apache.org/r/50653/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Masahiro Tanaka
> 
>

Re: Review Request 50653: AMBARI-17308 Ambari Logfeeder outputs a lot of errors due to parse date

2016-08-01 Thread Oliver Szabo 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50653/#review144326
---


Ship it!




Ship It!

- Oliver Szabo


On Aug. 1, 2016, 10:23 a.m., Masahiro Tanaka wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50653/
> ---
> 
> (Updated Aug. 1, 2016, 10:23 a.m.)
> 
> 
> Review request for Ambari, Daniel Gergely, Dmytro Sen, Oliver Szabo, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-17308
> https://issues.apache.org/jira/browse/AMBARI-17308
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> In logsearch_feeder service log, we got errors like below
> ```
> 2016-06-20 15:28:09,368 ERROR file=ambari-audit.log 
> org.apache.ambari.logfeeder.mapper.MapperDate LogFeederUtil.java:356 - Error 
> applying date transformation. isEpoch=false, 
> dateFormat=-MM-dd'T'HH:mm:ss.SSSZ, value=2016-06-20T15:28:08.000. 
> mapClass=map_date, input=input:source=file, 
> path=/var/log/ambari-server/ambari-audit.log, fieldName=logtime. Messages 
> suppressed before: 2
> java.text.ParseException: Unparseable date: "2016-06-20T15:28:08.000"
>   at java.text.DateFormat.parse(DateFormat.java:366)
>   at 
> org.apache.ambari.logfeeder.mapper.MapperDate.apply(MapperDate.java:83)
>   at org.apache.ambari.logfeeder.filter.Filter.apply(Filter.java:154)
>   at 
> org.apache.ambari.logfeeder.filter.FilterGrok.applyMessage(FilterGrok.java:291)
>   at 
> org.apache.ambari.logfeeder.filter.FilterGrok.flush(FilterGrok.java:320)
>   at org.apache.ambari.logfeeder.input.Input.flush(Input.java:125)
>   at 
> org.apache.ambari.logfeeder.input.InputFile.processFile(InputFile.java:430)
>   at org.apache.ambari.logfeeder.input.InputFile.start(InputFile.java:260)
>   at org.apache.ambari.logfeeder.input.Input.run(Input.java:100)
>   at java.lang.Thread.run(Thread.java:745)
> ```
> ambari-audit.log is like below
> ```
> 2016-07-21T01:52:49.875+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
> 2016-07-21T01:52:49.905+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu16/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu16), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu16/2.x/BUILDS/2.5.0.0-1025)
> 2016-07-21T01:52:50.015+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-UTILS-1.1.0.21),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-UTILS-1.1.0.21), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/ubuntu14)
> ```
> I think date format of the ambari-audit.log (2016-07-21T01:52:49.875+09) 
> should be like 2016-07-21T01:52:49.875+0900, since grok-pattern can't handle 
> 2016-07-21T01:52:49.875+09 format.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerDefaultImpl.java
>  ea6524f 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/input.config-ambari.json.j2
>  5f751bf 
> 
> Diff: https://reviews.apache.org/r/50653/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Masahiro Tanaka
> 
>

Review Request 50653: AMBARI-17308 Ambari Logfeeder outputs a lot of errors due to parse date

2016-08-01 Thread Masahiro Tanaka 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50653/
---

Review request for Ambari, Daniel Gergely, Dmytro Sen, Oliver Szabo, and 
Sebastian Toader.


Bugs: AMBARI-17308
https://issues.apache.org/jira/browse/AMBARI-17308


Repository: ambari


Description
---

In logsearch_feeder service log, we got errors like below
```
2016-06-20 15:28:09,368 ERROR file=ambari-audit.log 
org.apache.ambari.logfeeder.mapper.MapperDate LogFeederUtil.java:356 - Error 
applying date transformation. isEpoch=false, 
dateFormat=-MM-dd'T'HH:mm:ss.SSSZ, value=2016-06-20T15:28:08.000. 
mapClass=map_date, input=input:source=file, 
path=/var/log/ambari-server/ambari-audit.log, fieldName=logtime. Messages 
suppressed before: 2
java.text.ParseException: Unparseable date: "2016-06-20T15:28:08.000"
at java.text.DateFormat.parse(DateFormat.java:366)
at 
org.apache.ambari.logfeeder.mapper.MapperDate.apply(MapperDate.java:83)
at org.apache.ambari.logfeeder.filter.Filter.apply(Filter.java:154)
at 
org.apache.ambari.logfeeder.filter.FilterGrok.applyMessage(FilterGrok.java:291)
at 
org.apache.ambari.logfeeder.filter.FilterGrok.flush(FilterGrok.java:320)
at org.apache.ambari.logfeeder.input.Input.flush(Input.java:125)
at 
org.apache.ambari.logfeeder.input.InputFile.processFile(InputFile.java:430)
at org.apache.ambari.logfeeder.input.InputFile.start(InputFile.java:260)
at org.apache.ambari.logfeeder.input.Input.run(Input.java:100)
at java.lang.Thread.run(Thread.java:745)
```
ambari-audit.log is like below
```
2016-07-21T01:52:49.875+09, User(admin), RemoteIp(192.168.72.1), 
Operation(Repository update), RequestType(PUT), 
url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
 ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
id(HDP-2.5), Base 
URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
2016-07-21T01:52:49.905+09, User(admin), RemoteIp(192.168.72.1), 
Operation(Repository update), RequestType(PUT), 
url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu16/repositories/HDP-2.5),
 ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu16), Repo 
id(HDP-2.5), Base 
URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu16/2.x/BUILDS/2.5.0.0-1025)
2016-07-21T01:52:50.015+09, User(admin), RemoteIp(192.168.72.1), 
Operation(Repository update), RequestType(PUT), 
url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-UTILS-1.1.0.21),
 ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
id(HDP-UTILS-1.1.0.21), Base 
URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/ubuntu14)
```
I think date format of the ambari-audit.log (2016-07-21T01:52:49.875+09) should 
be like 2016-07-21T01:52:49.875+0900, since grok-pattern can't handle 
2016-07-21T01:52:49.875+09 format.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerDefaultImpl.java
 ea6524f 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/input.config-ambari.json.j2
 5f751bf 

Diff: https://reviews.apache.org/r/50653/diff/


Testing
---

mvn clean test


Thanks,

Masahiro Tanaka