[jira] [Comment Edited] (HDDS-98) Adding Ozone Manager Audit Log
[
https://issues.apache.org/jira/browse/HDDS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603432#comment-16603432
]
Nanda kumar edited comment on HDDS-98 at 9/4/18 6:40 PM:
-
+1, the patch looks good to me. Ran the tests locally, no failures.
I will commit this shortly.
{code:java}
HDDS: Tests run: 97, Failures: 0, Errors: 0, Skipped: 3
Ozone: Tests run: 283, Failures: 0, Errors: 0, Skipped: 19
OzoneFS: Tests run: 10, Failures: 0, Errors: 0, Skipped: 0
{code}
was (Author: nandakumar131):
Ran the tests locally, no failures. I will commit this shortly.
{code:java}
HDDS: Tests run: 97, Failures: 0, Errors: 0, Skipped: 3
Ozone: Tests run: 283, Failures: 0, Errors: 0, Skipped: 19
OzoneFS: Tests run: 10, Failures: 0, Errors: 0, Skipped: 0
{code}
> Adding Ozone Manager Audit Log
> --
>
> Key: HDDS-98
> URL: https://issues.apache.org/jira/browse/HDDS-98
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Dinesh Chitlangia
>Priority: Major
> Labels: Logging, audit
> Fix For: 0.2.1
>
> Attachments: HDDS-98.001.patch, HDDS-98.002.patch, HDDS-98.003.patch,
> HDDS-98.004.patch, HDDS-98.005.patch, HDDS-98.006.patch, HDDS-98.007.patch,
> HDDS-98.008.patch, audit.log, log4j2.properties
>
>
> This ticket is opened to add ozone manager's audit log.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-98) Adding Ozone Manager Audit Log
[
https://issues.apache.org/jira/browse/HDDS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603324#comment-16603324
]
Dinesh Chitlangia edited comment on HDDS-98 at 9/4/18 5:01 PM:
---
[~elek] Looks like the unit test run has failed.
{{ [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
17.75 s <<< FAILURE! - in
org.apache.hadoop.hdds.scm.pipeline.TestPipelineClose}}
{{ [ERROR] org.apache.hadoop.hdds.scm.pipeline.TestPipelineClose Time elapsed:
0.157 s <<< ERROR!}}
{{ java.lang.IllegalStateException: Shutdown in progress, cannot remove a
shutdownHook}}
This is unrelated to the patch.
was (Author: dineshchitlangia):
[~elek] Looks like the unit test run has failed.
[ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 17.75 s
<<< FAILURE! - in org.apache.hadoop.hdds.scm.pipeline.TestPipelineClose
[ERROR] org.apache.hadoop.hdds.scm.pipeline.TestPipelineClose Time elapsed:
0.157 s <<< ERROR!
java.lang.IllegalStateException: Shutdown in progress, cannot remove a
shutdownHook
This is unrelated to the patch.
> Adding Ozone Manager Audit Log
> --
>
> Key: HDDS-98
> URL: https://issues.apache.org/jira/browse/HDDS-98
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Dinesh Chitlangia
>Priority: Major
> Labels: Logging, audit
> Fix For: 0.2.1
>
> Attachments: HDDS-98.001.patch, HDDS-98.002.patch, HDDS-98.003.patch,
> HDDS-98.004.patch, HDDS-98.005.patch, HDDS-98.006.patch, HDDS-98.007.patch,
> HDDS-98.008.patch, audit.log, log4j2.properties
>
>
> This ticket is opened to add ozone manager's audit log.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-98) Adding Ozone Manager Audit Log
[ https://issues.apache.org/jira/browse/HDDS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596872#comment-16596872 ] Dinesh Chitlangia edited comment on HDDS-98 at 8/29/18 9:30 PM: [~anu], [~xyao] - Request you to please review the patch. cc: [~jnp] - your inputs have been incorporated. was (Author: dineshchitlangia): [~anu], [~xyao] - Request you to please review the patch. > Adding Ozone Manager Audit Log > -- > > Key: HDDS-98 > URL: https://issues.apache.org/jira/browse/HDDS-98 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Dinesh Chitlangia >Priority: Major > Labels: Logging, audit > Fix For: 0.2.1 > > Attachments: HDDS-98.001.patch, HDDS-98.002.patch, HDDS-98.003.patch, > HDDS-98.004.patch, HDDS-98.005.patch, audit.log, log4j2.properties > > > This ticket is opened to add ozone manager's audit log. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-98) Adding Ozone Manager Audit Log
[
https://issues.apache.org/jira/browse/HDDS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591724#comment-16591724
]
Dinesh Chitlangia edited comment on HDDS-98 at 8/24/18 2:20 PM:
[~jnp] Thank you for a detailed feedback.
In this implementation, we are using the StructuredDataMessage (conforms to RFC
5424) for logging.
Its basic structure is:
{quote}{{{color:#59afe1}type{color} {color:#654982}ID{color}
{color:#14892c}dataMap{color} {color:#f79232}message{color} throwable}}
{quote}
* type - Customer action types we have defined for ozone like CREATE_VOLUME,
CREATE_BUCKET etc
* ID - I can log a string "username IPAddress"
* dataMap - It is a key-value map object that we prepare for logging and can
thus add as many key-value pairs as want. So this addresses the extensibility
aspect. We cannot log username/IPAddress as key-value pair in the dataMap
because the dataMap will sort by key while printing and currently there is no
API to turn off sorting.
* message - any specific message we may want to pass, it can also be left
blank. We decided to pass this with only two values SUCCESS and FAILURE
depending upon the result. In future when we build a log parser, this will be
handy for querying various combinations.
* throwable - if any exception was thrown in the actual operation, we can log
it. This is totally configurable via properties file to log - x lines of
exception or error message or not log at all.
Thus, we end up with a log entry like the one below.
{quote}{{2018-08-24 04:57:28,685 | INFO | OMAudit |
{color:#59afe1}CREATE_VOLUME{color} [{color:#654982}xiaoyu/172.18.0.3
{color}{color:#14892c}admin="xiaoyu" creationTime="0" owner="xiaoyu"
quotaInBytes="1099511627776" volume="xmen"{color}]
{color:#f79232}SUCCESS{color} |}}
{quote}
> I can definitely change the "username/IP" to "username IP" and ensure this
> is at top level by logging it as ID.
> Extensibility has been addressed by {color:#14892c}dataMap{color}
> With this structure, the parsing will likely be easier as the actual data in
> enclosed in [ ]. Even if there were nested [ ] inside this, I feel it will
> still be easy to parse when we write a parser.
> Given the above structure, unfortunately, we won't be able to log like the
> example you provided
Please do let me know if there are further improvements that can be made and
thank you once again for a detailed feedback. :)
was (Author: dineshchitlangia):
[~jnp] Thank you for a detailed feedback.
In this implementation, we are using the StructuredDataMessage (conforms to RFC
5424) for logging.
Its basic structure is:
{quote}{{{color:#59afe1}type{color} {color:#654982}ID{color} dataMap
{color:#f79232}message{color} throwable}}
{quote}
* type - Customer action types we have defined for ozone like CREATE_VOLUME,
CREATE_BUCKET etc
* ID - I can log a string "username IPAddress"
* dataMap - It is a key-value map object that we prepare for logging and can
thus add as many key-value pairs as want. So this addresses the extensibility
aspect. We cannot log username/IPAddress as key-value pair in the dataMap
because the dataMap will sort by key while printing and currently there is no
API to turn off sorting.
* message - any specific message we may want to pass, it can also be left
blank. We decided to pass this with only two values SUCCESS and FAILURE
depending upon the result. In future when we build a log parser, this will be
handy for querying various combinations.
* throwable - if any exception was thrown in the actual operation, we can log
it. This is totally configurable via properties file to log - x lines of
exception or error message or not log at all.
Thus, we end up with a log entry like the one below.
{quote}{{2018-08-24 04:57:28,685 | INFO | OMAudit |
{color:#59afe1}CREATE_VOLUME{color} [{color:#654982}xiaoyu/172.18.0.3
{color}{color:#14892c}admin="xiaoyu" creationTime="0" owner="xiaoyu"
quotaInBytes="1099511627776" volume="xmen"{color}]
{color:#f79232}SUCCESS{color} |}}
{quote}
> I can definitely change the "username/IP" to "username IP" and ensure this
> is at top level by logging it as ID.
> Extensibility has been addressed by {color:#14892c}dataMap{color}
> With this structure, the parsing will likely be easier as the actual data in
> enclosed in [ ]. Even if there were nested [ ] inside this, I feel it will
> still be easy to parse when we write a parser.
> Given the above structure, unfortunately, we won't be able to log like the
> example you provided
Please do let me know if there are further improvements that can be made and
thank you once again for a detailed feedback. :)
> Adding Ozone Manager Audit Log
> --
>
> Key: HDDS-98
> URL: https://issues.apache.org/jira/browse/HDDS-98
> Project: Hadoop Distributed Data Store
> Issue
[jira] [Comment Edited] (HDDS-98) Adding Ozone Manager Audit Log
[
https://issues.apache.org/jira/browse/HDDS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591724#comment-16591724
]
Dinesh Chitlangia edited comment on HDDS-98 at 8/24/18 2:19 PM:
[~jnp] Thank you for a detailed feedback.
In this implementation, we are using the StructuredDataMessage (conforms to RFC
5424) for logging.
Its basic structure is:
{quote}{{{color:#59afe1}type{color} {color:#654982}ID{color} dataMap
{color:#f79232}message{color} throwable}}
{quote}
* type - Customer action types we have defined for ozone like CREATE_VOLUME,
CREATE_BUCKET etc
* ID - I can log a string "username IPAddress"
* dataMap - It is a key-value map object that we prepare for logging and can
thus add as many key-value pairs as want. So this addresses the extensibility
aspect. We cannot log username/IPAddress as key-value pair in the dataMap
because the dataMap will sort by key while printing and currently there is no
API to turn off sorting.
* message - any specific message we may want to pass, it can also be left
blank. We decided to pass this with only two values SUCCESS and FAILURE
depending upon the result. In future when we build a log parser, this will be
handy for querying various combinations.
* throwable - if any exception was thrown in the actual operation, we can log
it. This is totally configurable via properties file to log - x lines of
exception or error message or not log at all.
Thus, we end up with a log entry like the one below.
{quote}{{2018-08-24 04:57:28,685 | INFO | OMAudit |
{color:#59afe1}CREATE_VOLUME{color} [{color:#654982}xiaoyu/172.18.0.3
{color}{color:#14892c}admin="xiaoyu" creationTime="0" owner="xiaoyu"
quotaInBytes="1099511627776" volume="xmen"{color}]
{color:#f79232}SUCCESS{color} |}}
{quote}
> I can definitely change the "username/IP" to "username IP" and ensure this
> is at top level by logging it as ID.
> Extensibility has been addressed by {color:#14892c}dataMap{color}
> With this structure, the parsing will likely be easier as the actual data in
> enclosed in [ ]. Even if there were nested [ ] inside this, I feel it will
> still be easy to parse when we write a parser.
> Given the above structure, unfortunately, we won't be able to log like the
> example you provided
Please do let me know if there are further improvements that can be made and
thank you once again for a detailed feedback. :)
was (Author: dineshchitlangia):
[~jnp] Thank you for a detailed feedback.
In this implementation, we are using the StructuredDataMessage (conforms to RFC
5424) for logging.
Its basic structure is:
{quote}{{{color:#59afe1}type{color} {color:#654982}ID{color} dataMap
{color:#f79232}message{color} throwable}}
{quote}
* type - Customer action types we have defined for ozone like CREATE_VOLUME,
CREATE_BUCKET etc
* ID - I can log a string "username IPAddress"
* dataMap - It is a key-value map object that we prepare for logging and can
thus add as many key-value pairs as want. So this addresses the extensibility
aspect. We cannot log username/IPAddress as key-value pair in the dataMap
because the dataMap will sort by key while printing and currently there is no
API to turn off sorting.
* message - any specific message we may want to pass, it can also be left
blank. We decided to pass this with only two values SUCCESS and FAILURE
depending upon the result. In future when we build a log parser, this will be
handy for querying various combinations.
* throwable - if any exception was thrown in the actual operation, we can log
it. This is totally configurable via properties file to log - x lines of
exception or error message or not log at all.
Thus, we end up with a log entry like the one below.
{quote}{{2018-08-24 04:57:28,685 | INFO | OMAudit |
{color:#59afe1}CREATE_VOLUME{color} [{color:#654982}xiaoyu/172.18.0.3
{color}{color:#14892c}admin="xiaoyu" creationTime="0" owner="xiaoyu"
quotaInBytes="1099511627776" volume="xmen"{color}]
{color:#f79232}SUCCESS{color} |}}
{quote}
> I can definitely change the "username/IP" to "username IP" and ensure this
> is at top level by logging it as ID.
> Extensibility has been addressed by {color:#14892c}dataMap{color}
{color:#14892c}> With this structure, the parsing will likely be easier as the
actual data in enclosed in [ ]. Even if there were nested [ ] inside this, I
feel it will still be easy to parse when we write a parser{color}
> Given the above structure, unfortunately, we won't be able to log like the
> example you provided
Please do let me know if there are further improvements that can be made and
thank you once again for a detailed feedback. :)
> Adding Ozone Manager Audit Log
> --
>
> Key: HDDS-98
> URL: https://issues.apache.org/jira/browse/HDDS-98
> Project: Hadoop Distributed Data Store
> Issue
[jira] [Comment Edited] (HDDS-98) Adding Ozone Manager Audit Log
[
https://issues.apache.org/jira/browse/HDDS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591724#comment-16591724
]
Dinesh Chitlangia edited comment on HDDS-98 at 8/24/18 2:17 PM:
[~jnp] Thank you for a detailed feedback.
In this implementation, we are using the StructuredDataMessage (conforms to RFC
5424) for logging.
Its basic structure is:
{quote}{{{color:#59afe1}type{color} {color:#654982}ID{color} dataMap
{color:#f79232}message{color} throwable}}
{quote}
* type - Customer action types we have defined for ozone like CREATE_VOLUME,
CREATE_BUCKET etc
* ID - I can log a string "username IPAddress"
* dataMap - It is a key-value map object that we prepare for logging and can
thus add as many key-value pairs as want. So this addresses the extensibility
aspect. We cannot log username/IPAddress as key-value pair in the dataMap
because the dataMap will sort by key while printing and currently there is no
API to turn off sorting.
* message - any specific message we may want to pass, it can also be left
blank. We decided to pass this with only two values SUCCESS and FAILURE
depending upon the result. In future when we build a log parser, this will be
handy for querying various combinations.
* throwable - if any exception was thrown in the actual operation, we can log
it. This is totally configurable via properties file to log - x lines of
exception or error message or not log at all.
Thus, we end up with a log entry like the one below.
{quote}{{2018-08-24 04:57:28,685 | INFO | OMAudit |
{color:#59afe1}CREATE_VOLUME{color} [{color:#654982}xiaoyu/172.18.0.3
{color}{color:#14892c}admin="xiaoyu" creationTime="0" owner="xiaoyu"
quotaInBytes="1099511627776" volume="xmen"{color}]
{color:#f79232}SUCCESS{color} |}}
{quote}
> I can definitely change the "username/IP" to "username IP" and ensure this
> is at top level by logging it as ID.
> Extensibility has been addressed by {color:#14892c}dataMap{color}
{color:#14892c}> With this structure, the parsing will likely be easier as the
actual data in enclosed in [ ]. Even if there were nested [ ] inside this, I
feel it will still be easy to parse when we write a parser{color}
> Given the above structure, unfortunately, we won't be able to log like the
> example you provided
Please do let me know if there are further improvements that can be made and
thank you once again for a detailed feedback. :)
was (Author: dineshchitlangia):
[~jnp] Thank you for a detailed feedback.
In this implementation, we are using the StructuredDataMessage (conforms to RFC
5424) for logging.
Its basic structure is:
{quote}{{{color:#59afe1}type{color} {color:#654982}ID{color} dataMap
{color:#f79232}message{color} throwable}}
{quote} * type - Customer action types we have defined for ozone like
CREATE_VOLUME, CREATE_BUCKET etc
* ID - I can log a string "username IPAddress"
* dataMap - It is a key-value map object that we prepare for logging and can
thus add as many key-value pairs as want. So this addresses the extensibility
aspect. We cannot log username/IPAddress as key-value pair in the dataMap
because the dataMap will sort by key while printing and currently there is no
API to turn off sorting.
* message - any specific message we may want to pass, it can also be left
blank. We decided to pass this with only two values SUCCESS and FAILURE
depending upon the result. In future when we build a log parser, this will be
handy for querying various combinations.
* throwable - if any exception was thrown in the actual operation, we can log
it. This is totally configurable via properties file to log - x lines of
exception or error message or not log at all.
Thus, we end up with a log entry like the one below.
{quote}{{2018-08-24 04:57:28,685 | INFO | OMAudit |
{color:#59afe1}CREATE_VOLUME{color} [{color:#654982}xiaoyu/172.18.0.3
{color}{color:#14892c}admin="xiaoyu" creationTime="0" owner="xiaoyu"
quotaInBytes="1099511627776" volume="xmen"{color}]
{color:#f79232}SUCCESS{color} |}}
{quote}
> I can definitely change the "username/IP" to "username IP" and ensure this
> is at top level by logging it as ID.
> Extensibility has been addressed by {color:#14892c}dataMap{color}
{color:#14892c}> With this structure, the parsing will likely be easier as the
actual data in enclosed in [ ]. Even if there were nested [ ] inside this, I
feel it will still be easy to parse when we write a parser{color}
> Given the above structure, unfortunately, we won't be able to log like the
> example you provided
Please do let me know if there are further improvements that can be made and
thank you once again for a detailed feedback. :)
> Adding Ozone Manager Audit Log
> --
>
> Key: HDDS-98
> URL: https://issues.apache.org/jira/browse/HDDS-98
> Project: Hadoop Distributed Data Store
>
[jira] [Comment Edited] (HDDS-98) Adding Ozone Manager Audit Log
[
https://issues.apache.org/jira/browse/HDDS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591724#comment-16591724
]
Dinesh Chitlangia edited comment on HDDS-98 at 8/24/18 2:16 PM:
[~jnp] Thank you for a detailed feedback.
In this implementation, we are using the StructuredDataMessage (conforms to RFC
5424) for logging.
Its basic structure is:
{quote}{{{color:#59afe1}type{color} {color:#654982}ID{color} dataMap
{color:#f79232}message{color} throwable}}
{quote}
* type - Customer action types we have defined for ozone like CREATE_VOLUME,
CREATE_BUCKET etc
* ID - I can log a string "username IPAddress"
* dataMap - It is a key-value map object that we prepare for logging and can
thus add as many key-value pairs as want. So this addresses the extensibility
aspect. We cannot log username/IPAddress as key-value pair in the dataMap
because the dataMap will sort by key while printing and currently there is no
API to turn off sorting.
* message - any specific message we may want to pass, it can also be left
blank. We decided to pass this with only two values SUCCESS and FAILURE
depending upon the result. In future when we build a log parser, this will be
handy for querying various combinations.
* throwable - if any exception was thrown in the actual operation, we can log
it. This is totally configurable via properties file to log - x lines of
exception or error message or not log at all.
Thus, we end up with a log entry like the one below.
{quote}{{2018-08-24 04:57:28,685 | INFO | OMAudit |
{color:#59afe1}CREATE_VOLUME{color} [{color:#654982}xiaoyu/172.18.0.3
{color}{color:#14892c}admin="xiaoyu" creationTime="0" owner="xiaoyu"
quotaInBytes="1099511627776" volume="xmen"{color}]
{color:#f79232}SUCCESS{color} |}}
{quote}
> I can definitely change the "username/IP" to "username IP" and ensure this
> is at top level by logging it as ID.
> Extensibility has been addressed by {color:#14892c}dataMap{color}
{color:#14892c}> With this structure, the parsing will likely be easier as the
actual data in enclosed in [ ]. Even if there were nested [ ] inside this, I
feel it will still be easy to parse when we write a parser{color}
> Given the above structure, unfortunately, we won't be able to log like the
> example you provided
Please do let me know if there are further improvements that can be made and
thank you once again for a detailed feedback. :)
was (Author: dineshchitlangia):
[~jnp] Thank you for a detailed feedback.
In this implementation, we are using the StructuredDataMessage (conforms to RFC
5424) for logging.
Its basic structure is:
{quote}{{{color:#59afe1}type{color} {color:#654982}ID{color} dataMap
{color:#f79232}message{color} throwable}}
{quote} * type - Customer action types we have defined for ozone like
CREATE_VOLUME, CREATE_BUCKET etc
* ID - I can log a string "username IPAddress"
* dataMap - It is a key-value map object that we prepare for logging and can
thus add as many key-value pairs as want. So this addresses the extensibility
aspect.
* message - any specific message we may want to pass, it can also be left
blank. We decided to pass this with only two values SUCCESS and FAILURE
depending upon the result. In future when we build a log parser, this will be
handy for querying various combinations.
* throwable - if any exception was thrown in the actual operation, we can log
it. This is totally configurable via properties file to log - x lines of
exception or error message or not log at all.
Thus, we end up with a log entry like the one below.
{quote}{{2018-08-24 04:57:28,685 | INFO | OMAudit |
{color:#59afe1}CREATE_VOLUME{color} [{color:#654982}xiaoyu/172.18.0.3
{color}{color:#14892c}admin="xiaoyu" creationTime="0" owner="xiaoyu"
quotaInBytes="1099511627776" volume="xmen"{color}]
{color:#f79232}SUCCESS{color} |}}
{quote}
> I can definitely change the "username/IP" to "username IP" and ensure this
> is at top level by logging it as ID.
> Extensibility has been addressed by {color:#14892c}dataMap{color}
{color:#14892c}{color:#33}> With this structure, the parsing will likely be
easier as the actual data in enclosed in [ ]. Even if there were nested [ ]
inside this, I feel it will still be easy to parse when we write a parser{color}
{color}
{color:#14892c}{color:#33}> Given the above structure, unfortunately, we
won't be able to log like the example you provided{color}{color}
{color:#14892c}{color:#33}Please do let me know if there are further
improvements that can be made and thank you once again for a detailed feedback.
:){color}{color}
> Adding Ozone Manager Audit Log
> --
>
> Key: HDDS-98
> URL: https://issues.apache.org/jira/browse/HDDS-98
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Xiaoyu
[jira] [Comment Edited] (HDDS-98) Adding Ozone Manager Audit Log
[
https://issues.apache.org/jira/browse/HDDS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591724#comment-16591724
]
Dinesh Chitlangia edited comment on HDDS-98 at 8/24/18 2:16 PM:
[~jnp] Thank you for a detailed feedback.
In this implementation, we are using the StructuredDataMessage (conforms to RFC
5424) for logging.
Its basic structure is:
{quote}{{{color:#59afe1}type{color} {color:#654982}ID{color} dataMap
{color:#f79232}message{color} throwable}}
{quote} * type - Customer action types we have defined for ozone like
CREATE_VOLUME, CREATE_BUCKET etc
* ID - I can log a string "username IPAddress"
* dataMap - It is a key-value map object that we prepare for logging and can
thus add as many key-value pairs as want. So this addresses the extensibility
aspect. We cannot log username/IPAddress as key-value pair in the dataMap
because the dataMap will sort by key while printing and currently there is no
API to turn off sorting.
* message - any specific message we may want to pass, it can also be left
blank. We decided to pass this with only two values SUCCESS and FAILURE
depending upon the result. In future when we build a log parser, this will be
handy for querying various combinations.
* throwable - if any exception was thrown in the actual operation, we can log
it. This is totally configurable via properties file to log - x lines of
exception or error message or not log at all.
Thus, we end up with a log entry like the one below.
{quote}{{2018-08-24 04:57:28,685 | INFO | OMAudit |
{color:#59afe1}CREATE_VOLUME{color} [{color:#654982}xiaoyu/172.18.0.3
{color}{color:#14892c}admin="xiaoyu" creationTime="0" owner="xiaoyu"
quotaInBytes="1099511627776" volume="xmen"{color}]
{color:#f79232}SUCCESS{color} |}}
{quote}
> I can definitely change the "username/IP" to "username IP" and ensure this
> is at top level by logging it as ID.
> Extensibility has been addressed by {color:#14892c}dataMap{color}
{color:#14892c}> With this structure, the parsing will likely be easier as the
actual data in enclosed in [ ]. Even if there were nested [ ] inside this, I
feel it will still be easy to parse when we write a parser{color}
> Given the above structure, unfortunately, we won't be able to log like the
> example you provided
Please do let me know if there are further improvements that can be made and
thank you once again for a detailed feedback. :)
was (Author: dineshchitlangia):
[~jnp] Thank you for a detailed feedback.
In this implementation, we are using the StructuredDataMessage (conforms to RFC
5424) for logging.
Its basic structure is:
{quote}{{{color:#59afe1}type{color} {color:#654982}ID{color} dataMap
{color:#f79232}message{color} throwable}}
{quote} * type - Customer action types we have defined for ozone like
CREATE_VOLUME, CREATE_BUCKET etc
* ID - I can log a string "username IPAddress"
* dataMap - It is a key-value map object that we prepare for logging and can
thus add as many key-value pairs as want. So this addresses the extensibility
aspect. We cannot log username/IPAddress as key-value pair in the dataMap
because the dataMap will sort by key while printing and currently there is no
API to turn off sorting.
* message - any specific message we may want to pass, it can also be left
blank. We decided to pass this with only two values SUCCESS and FAILURE
depending upon the result. In future when we build a log parser, this will be
handy for querying various combinations.
* throwable - if any exception was thrown in the actual operation, we can log
it. This is totally configurable via properties file to log - x lines of
exception or error message or not log at all.
Thus, we end up with a log entry like the one below.
{quote}{{2018-08-24 04:57:28,685 | INFO | OMAudit |
{color:#59afe1}CREATE_VOLUME{color} [{color:#654982}xiaoyu/172.18.0.3
{color}{color:#14892c}admin="xiaoyu" creationTime="0" owner="xiaoyu"
quotaInBytes="1099511627776" volume="xmen"{color}]
{color:#f79232}SUCCESS{color} |}}
{quote}
> I can definitely change the "username/IP" to "username IP" and ensure this
> is at top level by logging it as ID.
> Extensibility has been addressed by {color:#14892c}dataMap{color}
{color:#14892c}> With this structure, the parsing will likely be easier as the
actual data in enclosed in [ ]. Even if there were nested [ ] inside this, I
feel it will still be easy to parse when we write a parser{color}
> Given the above structure, unfortunately, we won't be able to log like the
> example you provided
Please do let me know if there are further improvements that can be made and
thank you once again for a detailed feedback. :)
> Adding Ozone Manager Audit Log
> --
>
> Key: HDDS-98
> URL: https://issues.apache.org/jira/browse/HDDS-98
> Project: Hadoop Distributed Data Store
>
[jira] [Comment Edited] (HDDS-98) Adding Ozone Manager Audit Log
[
https://issues.apache.org/jira/browse/HDDS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591724#comment-16591724
]
Dinesh Chitlangia edited comment on HDDS-98 at 8/24/18 2:16 PM:
[~jnp] Thank you for a detailed feedback.
In this implementation, we are using the StructuredDataMessage (conforms to RFC
5424) for logging.
Its basic structure is:
{quote}{{{color:#59afe1}type{color} {color:#654982}ID{color} dataMap
{color:#f79232}message{color} throwable}}
{quote} * type - Customer action types we have defined for ozone like
CREATE_VOLUME, CREATE_BUCKET etc
* ID - I can log a string "username IPAddress"
* dataMap - It is a key-value map object that we prepare for logging and can
thus add as many key-value pairs as want. So this addresses the extensibility
aspect. We cannot log username/IPAddress as key-value pair in the dataMap
because the dataMap will sort by key while printing and currently there is no
API to turn off sorting.
* message - any specific message we may want to pass, it can also be left
blank. We decided to pass this with only two values SUCCESS and FAILURE
depending upon the result. In future when we build a log parser, this will be
handy for querying various combinations.
* throwable - if any exception was thrown in the actual operation, we can log
it. This is totally configurable via properties file to log - x lines of
exception or error message or not log at all.
Thus, we end up with a log entry like the one below.
{quote}{{2018-08-24 04:57:28,685 | INFO | OMAudit |
{color:#59afe1}CREATE_VOLUME{color} [{color:#654982}xiaoyu/172.18.0.3
{color}{color:#14892c}admin="xiaoyu" creationTime="0" owner="xiaoyu"
quotaInBytes="1099511627776" volume="xmen"{color}]
{color:#f79232}SUCCESS{color} |}}
{quote}
> I can definitely change the "username/IP" to "username IP" and ensure this
> is at top level by logging it as ID.
> Extensibility has been addressed by {color:#14892c}dataMap{color}
{color:#14892c}> With this structure, the parsing will likely be easier as the
actual data in enclosed in [ ]. Even if there were nested [ ] inside this, I
feel it will still be easy to parse when we write a parser{color}
> Given the above structure, unfortunately, we won't be able to log like the
> example you provided
Please do let me know if there are further improvements that can be made and
thank you once again for a detailed feedback. :)
was (Author: dineshchitlangia):
[~jnp] Thank you for a detailed feedback.
In this implementation, we are using the StructuredDataMessage (conforms to RFC
5424) for logging.
Its basic structure is:
{quote}{{{color:#59afe1}type{color} {color:#654982}ID{color} dataMap
{color:#f79232}message{color} throwable}}
{quote} * type - Customer action types we have defined for ozone like
CREATE_VOLUME, CREATE_BUCKET etc
* ID - I can log a string "username IPAddress"
* dataMap - It is a key-value map object that we prepare for logging and can
thus add as many key-value pairs as want. So this addresses the extensibility
aspect. We cannot log username/IPAddress as key-value pair in the dataMap
because the dataMap will sort by key while printing and currently there is no
API to turn off sorting.
* message - any specific message we may want to pass, it can also be left
blank. We decided to pass this with only two values SUCCESS and FAILURE
depending upon the result. In future when we build a log parser, this will be
handy for querying various combinations.
* throwable - if any exception was thrown in the actual operation, we can log
it. This is totally configurable via properties file to log - x lines of
exception or error message or not log at all.
Thus, we end up with a log entry like the one below.
{quote}{{2018-08-24 04:57:28,685 | INFO | OMAudit |
{color:#59afe1}CREATE_VOLUME{color} [{color:#654982}xiaoyu/172.18.0.3
{color}{color:#14892c}admin="xiaoyu" creationTime="0" owner="xiaoyu"
quotaInBytes="1099511627776" volume="xmen"{color}]
{color:#f79232}SUCCESS{color} |}}
{quote}
> I can definitely change the "username/IP" to "username IP" and ensure this
> is at top level by logging it as ID.
> Extensibility has been addressed by {color:#14892c}dataMap{color}
{color:#14892c}> With this structure, the parsing will likely be easier as the
actual data in enclosed in [ ]. Even if there were nested [ ] inside this, I
feel it will still be easy to parse when we write a parser{color}
> Given the above structure, unfortunately, we won't be able to log like the
> example you provided
Please do let me know if there are further improvements that can be made and
thank you once again for a detailed feedback. :)
> Adding Ozone Manager Audit Log
> --
>
> Key: HDDS-98
> URL: https://issues.apache.org/jira/browse/HDDS-98
> Project: Hadoop Distributed Data Store
>
[jira] [Comment Edited] (HDDS-98) Adding Ozone Manager Audit Log
[
https://issues.apache.org/jira/browse/HDDS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591724#comment-16591724
]
Dinesh Chitlangia edited comment on HDDS-98 at 8/24/18 2:16 PM:
[~jnp] Thank you for a detailed feedback.
In this implementation, we are using the StructuredDataMessage (conforms to RFC
5424) for logging.
Its basic structure is:
{quote}{{{color:#59afe1}type{color} {color:#654982}ID{color} dataMap
{color:#f79232}message{color} throwable}}
{quote} * type - Customer action types we have defined for ozone like
CREATE_VOLUME, CREATE_BUCKET etc
* ID - I can log a string "username IPAddress"
* dataMap - It is a key-value map object that we prepare for logging and can
thus add as many key-value pairs as want. So this addresses the extensibility
aspect. We cannot log username/IPAddress as key-value pair in the dataMap
because the dataMap will sort by key while printing and currently there is no
API to turn off sorting.
* message - any specific message we may want to pass, it can also be left
blank. We decided to pass this with only two values SUCCESS and FAILURE
depending upon the result. In future when we build a log parser, this will be
handy for querying various combinations.
* throwable - if any exception was thrown in the actual operation, we can log
it. This is totally configurable via properties file to log - x lines of
exception or error message or not log at all.
Thus, we end up with a log entry like the one below.
{quote}{{2018-08-24 04:57:28,685 | INFO | OMAudit |
{color:#59afe1}CREATE_VOLUME{color} [{color:#654982}xiaoyu/172.18.0.3
{color}{color:#14892c}admin="xiaoyu" creationTime="0" owner="xiaoyu"
quotaInBytes="1099511627776" volume="xmen"{color}]
{color:#f79232}SUCCESS{color} |}}
{quote}
> I can definitely change the "username/IP" to "username IP" and ensure this
> is at top level by logging it as ID.
> Extensibility has been addressed by {color:#14892c}dataMap{color}
{color:#14892c}> With this structure, the parsing will likely be easier as the
actual data in enclosed in [ ]. Even if there were nested [ ] inside this, I
feel it will still be easy to parse when we write a parser{color}
> Given the above structure, unfortunately, we won't be able to log like the
> example you provided
Please do let me know if there are further improvements that can be made and
thank you once again for a detailed feedback. :)
was (Author: dineshchitlangia):
[~jnp] Thank you for a detailed feedback.
In this implementation, we are using the StructuredDataMessage (conforms to RFC
5424) for logging.
Its basic structure is:
{quote}{{{color:#59afe1}type{color} {color:#654982}ID{color} dataMap
{color:#f79232}message{color} throwable}}
{quote}
* type - Customer action types we have defined for ozone like CREATE_VOLUME,
CREATE_BUCKET etc
* ID - I can log a string "username IPAddress"
* dataMap - It is a key-value map object that we prepare for logging and can
thus add as many key-value pairs as want. So this addresses the extensibility
aspect. We cannot log username/IPAddress as key-value pair in the dataMap
because the dataMap will sort by key while printing and currently there is no
API to turn off sorting.
* message - any specific message we may want to pass, it can also be left
blank. We decided to pass this with only two values SUCCESS and FAILURE
depending upon the result. In future when we build a log parser, this will be
handy for querying various combinations.
* throwable - if any exception was thrown in the actual operation, we can log
it. This is totally configurable via properties file to log - x lines of
exception or error message or not log at all.
Thus, we end up with a log entry like the one below.
{quote}{{2018-08-24 04:57:28,685 | INFO | OMAudit |
{color:#59afe1}CREATE_VOLUME{color} [{color:#654982}xiaoyu/172.18.0.3
{color}{color:#14892c}admin="xiaoyu" creationTime="0" owner="xiaoyu"
quotaInBytes="1099511627776" volume="xmen"{color}]
{color:#f79232}SUCCESS{color} |}}
{quote}
> I can definitely change the "username/IP" to "username IP" and ensure this
> is at top level by logging it as ID.
> Extensibility has been addressed by {color:#14892c}dataMap{color}
{color:#14892c}> With this structure, the parsing will likely be easier as the
actual data in enclosed in [ ]. Even if there were nested [ ] inside this, I
feel it will still be easy to parse when we write a parser{color}
> Given the above structure, unfortunately, we won't be able to log like the
> example you provided
Please do let me know if there are further improvements that can be made and
thank you once again for a detailed feedback. :)
> Adding Ozone Manager Audit Log
> --
>
> Key: HDDS-98
> URL: https://issues.apache.org/jira/browse/HDDS-98
> Project: Hadoop Distributed Data Store
>
[jira] [Comment Edited] (HDDS-98) Adding Ozone Manager Audit Log
[
https://issues.apache.org/jira/browse/HDDS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590899#comment-16590899
]
Dinesh Chitlangia edited comment on HDDS-98 at 8/24/18 5:03 AM:
[~jnp] Thank you for the feedback.
{quote}The username executing the command should be a top level field.
{quote}
Yes, I am currently working on this.
I propose the following format:
{color:#654982}/172.1.1.1{color} followed by the rest of the information
being logged for the said action.
This is to avoid conflict with username field in some of the actions like
createVolume where VolumeArgs will contain the similar field which may not
necessarily be the same as the remoteUser making that call.
Also, since we are using StructuredDataMessage for logging, the id cannot
exceed 32 characters. Thus by using the format {color:#654982}remote
/172.1.1.1{color} we can restrict the id under 32 characters.
{quote}We may not need INFO/DEBUG information because audit logs are usually
controlled operations that should be recorded, and not by log levels.
{quote}
So, for the write/read success events, we can log at Level ALL and for failures
we can log at ERROR.
Let me know if the above proposals sound good.
Here are the sample logs from this approach:
{{2018-08-24 04:57:28,685 | INFO | OMAudit | CREATE_VOLUME
[{color:#654982}xiaoyu/172.18.0.3{color} admin="xiaoyu" creationTime="0"
owner="xiaoyu" quotaInBytes="1099511627776" volume="xmen"] SUCCESS |}}
{{2018-08-24 04:59:20,312 | INFO | OMAudit | CREATE_VOLUME
[{color:#654982}anu/172.18.0.2{color} admin="anu" creationTime="0" owner="anu"
quotaInBytes="8796093022208" volume="don"] SUCCESS |}}
was (Author: dineshchitlangia):
[~jnp] Thank you for the feedback.
{quote}The username executing the command should be a top level field.
{quote}
Yes, I am currently working on this.
I propose the following format:
{color:#654982}/172.1.1.1{color} followed by the rest of the information
being logged for the said action.
This is to avoid conflict with username field in some of the actions like
createVolume where VolumeArgs will contain the similar field which may not
necessarily be the same as the remoteUser making that call.
Also, since we are using StructuredDataMessage for logging, the id cannot
exceed 32 characters. Thus by using the format {color:#654982}remote
/172.1.1.1{color} we can restrict the id under 32 characters.
{quote}We may not need INFO/DEBUG information because audit logs are usually
controlled operations that should be recorded, and not by log levels.
{quote}
So, for the write/read success events, we can log at Level ALL and for failures
we can log at ERROR.
Let me know if the above proposals sound good.
> Adding Ozone Manager Audit Log
> --
>
> Key: HDDS-98
> URL: https://issues.apache.org/jira/browse/HDDS-98
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Dinesh Chitlangia
>Priority: Major
> Labels: Logging, audit
> Fix For: 0.2.1
>
> Attachments: HDDS-98.001.patch, HDDS-98.002.patch, HDDS-98.003.patch,
> audit.log, log4j2.properties
>
>
> This ticket is opened to add ozone manager's audit log.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-98) Adding Ozone Manager Audit Log
[
https://issues.apache.org/jira/browse/HDDS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590899#comment-16590899
]
Dinesh Chitlangia edited comment on HDDS-98 at 8/24/18 4:46 AM:
[~jnp] Thank you for the feedback.
{quote}The username executing the command should be a top level field.
{quote}
Yes, I am currently working on this.
I propose the following format:
{color:#654982}/172.1.1.1{color} followed by the rest of the information
being logged for the said action.
This is to avoid conflict with username field in some of the actions like
createVolume where VolumeArgs will contain the similar field which may not
necessarily be the same as the remoteUser making that call.
Also, since we are using StructuredDataMessage for logging, the id cannot
exceed 32 characters. Thus by using the format {color:#654982}remote
/172.1.1.1{color} we can restrict the id under 32 characters.
{quote}We may not need INFO/DEBUG information because audit logs are usually
controlled operations that should be recorded, and not by log levels.
{quote}
So, for the write/read success events, we can log at Level ALL and for failures
we can log at ERROR.
Let me know if the above proposals sound good.
was (Author: dineshchitlangia):
[~jnp] Thank you for the feedback.
{quote}The username executing the command should be a top level field.
{quote}
Yes, I am currently working on this.
I propose the following format:
{color:#654982}remote User/IP="/172.1.1.1"{color} followed by the rest of
the information being logged for the said action.
This is to avoid conflict with username field in some of the actions like
createVolume where VolumeArgs will contain the similar field which may not
necessarily be the same as the remoteUser making that call.
Also, since we are using StructuredDataMessage for logging, the id cannot
exceed 32 characters. Thus by using the format {color:#654982}remote
User/IP="/172.1.1.1"{color} we can restrict the id under 32 characters.
{quote}We may not need INFO/DEBUG information because audit logs are usually
controlled operations that should be recorded, and not by log levels.
{quote}
So, for the write/read success events, we can log at Level ALL and for failures
we can log at ERROR.
Let me know if the above proposals sound good.
> Adding Ozone Manager Audit Log
> --
>
> Key: HDDS-98
> URL: https://issues.apache.org/jira/browse/HDDS-98
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Dinesh Chitlangia
>Priority: Major
> Labels: Logging, audit
> Fix For: 0.2.1
>
> Attachments: HDDS-98.001.patch, HDDS-98.002.patch, HDDS-98.003.patch,
> audit.log, log4j2.properties
>
>
> This ticket is opened to add ozone manager's audit log.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-98) Adding Ozone Manager Audit Log
[
https://issues.apache.org/jira/browse/HDDS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590899#comment-16590899
]
Dinesh Chitlangia edited comment on HDDS-98 at 8/24/18 4:36 AM:
[~jnp] Thank you for the feedback.
{quote}The username executing the command should be a top level field.
{quote}
Yes, I am currently working on this.
I propose the following format:
{color:#654982}remote User/IP="/172.1.1.1"{color} followed by the rest of
the information being logged for the said action.
This is to avoid conflict with username field in some of the actions like
createVolume where VolumeArgs will contain the similar field which may not
necessarily be the same as the remoteUser making that call.
Also, since we are using StructuredDataMessage for logging, the id cannot
exceed 32 characters. Thus by using the format {color:#654982}remote
User/IP="/172.1.1.1"{color} we can restrict the id under 32 characters.
{quote}We may not need INFO/DEBUG information because audit logs are usually
controlled operations that should be recorded, and not by log levels.
{quote}
So, for the write/read success events, we can log at Level ALL and for failures
we can log at ERROR.
Let me know if the above proposals sound good.
was (Author: dineshchitlangia):
[~jnp] Thank you for the feedback.
{quote}The username executing the command should be a top level field.
{quote}
Yes, I am currently working on this.
I propose the following format:
remoteUser="" remoteIP="a.b.c.d" .. the rest of the information being
logged for the said action.
This is to avoid conflict with username field in some of the actions like
createVolume where VolumeArgs will contain the similar field which may not
necessarily be the same as the remoteUser making that call.
{quote}We may not need INFO/DEBUG information because audit logs are usually
controlled operations that should be recorded, and not by log levels.
{quote}
So, for the write/read success events, we can log at Level ALL and for failures
we can log at ERROR.
Let me know if the above proposals sound good.
> Adding Ozone Manager Audit Log
> --
>
> Key: HDDS-98
> URL: https://issues.apache.org/jira/browse/HDDS-98
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Dinesh Chitlangia
>Priority: Major
> Labels: Logging, audit
> Fix For: 0.2.1
>
> Attachments: HDDS-98.001.patch, HDDS-98.002.patch, HDDS-98.003.patch,
> audit.log, log4j2.properties
>
>
> This ticket is opened to add ozone manager's audit log.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-98) Adding Ozone Manager Audit Log
[ https://issues.apache.org/jira/browse/HDDS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588079#comment-16588079 ] Jitendra Nath Pandey edited comment on HDDS-98 at 8/21/18 10:24 PM: The username executing the command should be a top level field. We may not need INFO/DEBUG information because audit logs are usually controlled operations that should be recorded, and not by log levels. was (Author: jnp): The username executing the command should be a top level field. > Adding Ozone Manager Audit Log > -- > > Key: HDDS-98 > URL: https://issues.apache.org/jira/browse/HDDS-98 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Dinesh Chitlangia >Priority: Major > Labels: Logging, audit > Fix For: 0.2.1 > > Attachments: HDDS-98.001.patch, HDDS-98.002.patch, HDDS-98.003.patch, > audit.log, log4j2.properties > > > This ticket is opened to add ozone manager's audit log. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-98) Adding Ozone Manager Audit Log
[ https://issues.apache.org/jira/browse/HDDS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587104#comment-16587104 ] Dinesh Chitlangia edited comment on HDDS-98 at 8/21/18 7:35 AM: [~xyao] Please refer [^HDDS-98.002.patch] You can apply this patch to your local and once you have built it, you can untar and test ozone locally. I was able to run freon and generate some logs. I was also able to see the logs when in docker after manually started ozone on the datanode container. There are some issues with my docker, I need to uninstall and reinstall it. I have also added configuration to roll over the logs based on size/timeInterval Looking forward to your feedback/comments for any revision. was (Author: dineshchitlangia): [~xyao] Please refer HDDS-98.002.patch. You can apply this patch to your local and once you have built it, you can untar and test ozone locally. I was able to run freon and generate some logs. I was also able to see the logs when in docker after manually started ozone on the datanode container. There are some issues with my docker, I need to uninstall and reinstall it. I have also added configuration to roll over the logs based on size/timeInterval Looking forward to your feedback/comments for any revision. > Adding Ozone Manager Audit Log > -- > > Key: HDDS-98 > URL: https://issues.apache.org/jira/browse/HDDS-98 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Dinesh Chitlangia >Priority: Major > Labels: Logging, audit > Fix For: 0.2.1 > > Attachments: HDDS-98.001.patch, HDDS-98.002.patch, audit.log, > log4j2.properties > > > This ticket is opened to add ozone manager's audit log. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-98) Adding Ozone Manager Audit Log
[ https://issues.apache.org/jira/browse/HDDS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575282#comment-16575282 ] Dinesh Chitlangia edited comment on HDDS-98 at 8/13/18 8:48 PM: [~xyao] - I started implementing the changes in OzoneManager as discussed. I see certain roadblocks: # In methods like setOwner or setQuota we don't have access to the old owner or old quota values. Ideally, we would want to log old and new values in audit log # In methods like createVolume we will not have a valid value for creationTime since that is populated in the underlying implementation layer. Could you please share your thoughts on this? cc: [~anu] was (Author: dineshchitlangia): [~xyao] - I started implementing the changes in OzoneManager as discussed. I see certain roadblocks: # In methods like setOwner or setQuota we don't have access to the old owner or old quota values. Ideally, we would want to log old and new values in audit log # In methods like createVolume we will not have a valid value for creationTime since that is populated in the underlying implementation layer. Could you please share your thoughts on this? > Adding Ozone Manager Audit Log > -- > > Key: HDDS-98 > URL: https://issues.apache.org/jira/browse/HDDS-98 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Dinesh Chitlangia >Priority: Major > > This ticket is opened to add ozone manager's audit log. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-98) Adding Ozone Manager Audit Log
[ https://issues.apache.org/jira/browse/HDDS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16487642#comment-16487642 ] Xiaoyu Yao edited comment on HDDS-98 at 5/24/18 9:53 PM: - Initial Draft Proposal: 1. OM common audit log format similar to hdfs namenode audit log. Result | User | IP Address | Operation | Source Object (Volume/Bucket/Key) | Destination Object (Volume/Bucket/Key) | Object Info (for operations that changes object metadata such as permission, owner, etc) 2. Support default logger that logs to local file system. 3. Support multiple pluggable loggers to receive audit log for external management applications (like Ranger) to receive audit log stream. 4. Support async audit log 5. Selective audit log, e.g, allow specifying operations(only write) to audit log. Refer HDFS-3680 for HDFS audit log implementation details. was (Author: xyao): Initial Draft Proposal: 1. OM common audit log format similar to hdfs namenode audit log. Result | User | IP Address | Operation | Source Object (Volume/Bucket/Key) | Destination Object (Volume/Bucket/Key) | Object Info (for operations that changes object metadata such as permission, owner, etc) 2. Support default logger that logs to local file system. 3. Support multiple pluggable loggers to receive audit log for external management applications (like Ranger) to receive audit log stream. 4. Support async audit log 5. Selective audit log, e.g, allow specifying operations(only write) to audit log. Refer HDFS-3680 for HDFS audit log implementation details. > Adding Ozone Manager Audit Log > -- > > Key: HDDS-98 > URL: https://issues.apache.org/jira/browse/HDDS-98 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Dinesh Chitlangia >Priority: Major > > This ticket is opened to add ozone manager's audit log. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-98) Adding Ozone Manager Audit Log
[ https://issues.apache.org/jira/browse/HDDS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16487642#comment-16487642 ] Xiaoyu Yao edited comment on HDDS-98 at 5/24/18 9:50 PM: - Initial Draft Proposal: 1. OM common audit log format similar to hdfs namenode audit log. Result | User | IP Address | Operation | Source Object (Volume/Bucket/Key) | Destination Object (Volume/Bucket/Key) | Object Info (for operations that changes object metadata such as permission, owner, etc) 2. Support default logger that logs to local file system. 3. Support multiple pluggable loggers to receive audit log for external management applications (like Ranger) to receive audit log stream. 4. Support async audit log 5. Selective audit log, e.g, allow specifying operations(only write) to audit log. Refer HDFS-3680 for HDFS audit log implementation details. was (Author: xyao): Initial Draft Proposal: 1. OM common audit log format similar to hdfs namenode audit log. Result | User | IP Address | Operation | Source Object (Volume/Bucket/Key) | Destination Object (Volume/Bucket/Key) | Object Info (for operations that changes object metadata such as permission, owner, etc) 2. Support default logger that logs to local file system. 3. Support multiple pluggable loggers to receive audit log for external management applications (like Ranger) to receive audit log stream. 4. Support async audit log 5. Selective audit log, e.g, allow specifying operations(write) to audit log. Refer HDFS-3680 for HDFS audit log implementation details. > Adding Ozone Manager Audit Log > -- > > Key: HDDS-98 > URL: https://issues.apache.org/jira/browse/HDDS-98 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Dinesh Chitlangia >Priority: Major > > This ticket is opened to add ozone manager's audit log. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-98) Adding Ozone Manager Audit Log
[ https://issues.apache.org/jira/browse/HDDS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16487642#comment-16487642 ] Xiaoyu Yao edited comment on HDDS-98 at 5/24/18 9:50 PM: - Initial Draft Proposal: 1. OM common audit log format similar to hdfs namenode audit log. Result | User | IP Address | Operation | Source Object (Volume/Bucket/Key) | Destination Object (Volume/Bucket/Key) | Object Info (for operations that changes object metadata such as permission, owner, etc) 2. Support default logger that logs to local file system. 3. Support multiple pluggable loggers to receive audit log for external management applications (like Ranger) to receive audit log stream. 4. Support async audit log 5. Selective audit log, e.g, allow specifying operations(write) to audit log. Refer HDFS-3680 for HDFS audit log implementation details. was (Author: xyao): Initial Draft Proposal: 1. OM common audit log format similar to hdfs namenode audit log. Result | User | IP Address | Operation | Source Object (Volume/Bucket/Key) | Destination Object (Volume/Bucket/Key) | Object Info (for operations that changes object metadata such as permission, owner, etc) 2. Support default logger that logs to local file system. 3. Support multiple pluggable loggers to receive audit log for external management applications (like Ranger) to receive audit log stream. 4. Support async audit log Refer HDFS-3680 for HDFS audit log implementation details. > Adding Ozone Manager Audit Log > -- > > Key: HDDS-98 > URL: https://issues.apache.org/jira/browse/HDDS-98 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Dinesh Chitlangia >Priority: Major > > This ticket is opened to add ozone manager's audit log. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-98) Adding Ozone Manager Audit Log
[ https://issues.apache.org/jira/browse/HDDS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16487642#comment-16487642 ] Xiaoyu Yao edited comment on HDDS-98 at 5/24/18 9:41 PM: - Initial Draft Proposal: 1. OM common audit log format similar to hdfs namenode audit log. Result | User | IP Address | Operation | Source Object (Volume/Bucket/Key) | Destination Object (Volume/Bucket/Key) | Object Info (for operations that changes object metadata such as permission, owner, etc) 2. Support default logger that logs to local file system. 3. Support multiple pluggable loggers to receive audit log for external management applications (like Ranger) to receive audit log stream. 4. Support async audit log Refer HDFS-3680 for HDFS audit log implementation details. was (Author: xyao): Initial Draft Proposal: 1. OM common audit log format similar to hdfs namenode audit log. Result | User | IP Address | Operation | Source Object (Volume/Bucket/Key) | Destination Object (Volume/Bucket/Key) | Object Info (for operations that changes object metadata such as permission, owner, etc) 2. Support default logger that logs to local file system. 3. Support multiple pluggable loggers to receive audit log for external management applications (like Ranger) to receive audit log stream. Refer HDFS-3680 for HDFS audit log implementation details. > Adding Ozone Manager Audit Log > -- > > Key: HDDS-98 > URL: https://issues.apache.org/jira/browse/HDDS-98 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Dinesh Chitlangia >Priority: Major > > This ticket is opened to add ozone manager's audit log. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
