Repository: logging-log4j-audit Updated Branches: refs/heads/master fe0be8fb7 -> 43e9de642
Update site. Add missing license Project: http://git-wip-us.apache.org/repos/asf/logging-log4j-audit/repo Commit: http://git-wip-us.apache.org/repos/asf/logging-log4j-audit/commit/af897a5b Tree: http://git-wip-us.apache.org/repos/asf/logging-log4j-audit/tree/af897a5b Diff: http://git-wip-us.apache.org/repos/asf/logging-log4j-audit/diff/af897a5b Branch: refs/heads/master Commit: af897a5b797c46e9894299a283bf48e5956048bc Parents: fe0be8f Author: Ralph Goers <[email protected]> Authored: Sun Jun 10 09:52:59 2018 -0700 Committer: Ralph Goers <[email protected]> Committed: Sun Jun 10 09:52:59 2018 -0700 ---------------------------------------------------------------------- .../logging/log4j/audit/dto/AuditDto.java | 3 +- .../src/main/resources/application.properties | 15 ++++ src/site/markdown/catalog.md.vm | 5 +- src/site/markdown/download.md.vm | 37 +++----- src/site/markdown/index.md.vm | 20 ++++- src/site/markdown/requestContext.md.vm | 12 ++- src/site/site.xml | 2 +- src/site/xdoc/gettingStarted.xml.vm | 93 +++++++++++++++++++- 8 files changed, 152 insertions(+), 35 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/logging-log4j-audit/blob/af897a5b/log4j-audit/log4j-audit-api/src/main/java/org/apache/logging/log4j/audit/dto/AuditDto.java ---------------------------------------------------------------------- diff --git a/log4j-audit/log4j-audit-api/src/main/java/org/apache/logging/log4j/audit/dto/AuditDto.java b/log4j-audit/log4j-audit-api/src/main/java/org/apache/logging/log4j/audit/dto/AuditDto.java index 4830658..e9ae695 100644 --- a/log4j-audit/log4j-audit-api/src/main/java/org/apache/logging/log4j/audit/dto/AuditDto.java +++ b/log4j-audit/log4j-audit-api/src/main/java/org/apache/logging/log4j/audit/dto/AuditDto.java @@ -33,7 +33,8 @@ public class AuditDto { private String catalogId; /** - * The RequestContext Map. + * The RequestContext Map. This may be used as an alternative to passing the RequestContext as + * headers. */ private Map<String, String> requestContextMap; http://git-wip-us.apache.org/repos/asf/logging-log4j-audit/blob/af897a5b/log4j-catalog/log4j-catalog-editor/src/main/resources/application.properties ---------------------------------------------------------------------- diff --git a/log4j-catalog/log4j-catalog-editor/src/main/resources/application.properties b/log4j-catalog/log4j-catalog-editor/src/main/resources/application.properties index e9e2b8a..71a2d1d 100644 --- a/log4j-catalog/log4j-catalog-editor/src/main/resources/application.properties +++ b/log4j-catalog/log4j-catalog-editor/src/main/resources/application.properties @@ -1,3 +1,18 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + spring.thymeleaf.cache=false spring.thymeleaf.enabled=true spring.thymeleaf.prefix=classpath:/templates/ http://git-wip-us.apache.org/repos/asf/logging-log4j-audit/blob/af897a5b/src/site/markdown/catalog.md.vm ---------------------------------------------------------------------- diff --git a/src/site/markdown/catalog.md.vm b/src/site/markdown/catalog.md.vm index 40515d2..24d4196 100644 --- a/src/site/markdown/catalog.md.vm +++ b/src/site/markdown/catalog.md.vm @@ -22,9 +22,8 @@ $h1 The Apache Log4j Audit Catalog -Log4j-Audit uses a Catalog that describes the attributes and events that can be logged. The events may be grouped -by Products and/or Categories, but at this time nothing in Log4j Audit makes use of the product or catalog -definitions. The catalog is required to generate the audit events and to validate events being logged. +Log4j-Audit uses a Catalog that describes the attributes and events that can be logged. The catalog is required +to generate the audit events and to validate events being logged. $h2 The Catalog and Catalog Editor http://git-wip-us.apache.org/repos/asf/logging-log4j-audit/blob/af897a5b/src/site/markdown/download.md.vm ---------------------------------------------------------------------- diff --git a/src/site/markdown/download.md.vm b/src/site/markdown/download.md.vm index 6d4ad64..a206624 100644 --- a/src/site/markdown/download.md.vm +++ b/src/site/markdown/download.md.vm @@ -33,18 +33,18 @@ the originals on the main distribution server. | Apache Log4j Audit source (tar.gz) | [apache-log4j-audit-${Log4jAuditVersion}-src.tar.gz][srctar] | [apache-log4j-audit-${Log4jAuditVersion}-src.tar.gz.md5][srctarmd5] | [apache-log4j-audit-${Log4jAuditVersion}-src.tar.gz.asc][srctarasc] | | Apache Log4j Audit source (zip) | [apache-log4j-audit-${Log4jAuditVersion}-src.zip][srczip] | [apache-log4j-audit-${Log4jAuditVersion}-src.zip.md5][srczipmd5] | [apache-log4j-audit-${Log4jAuditVersion}-src.zip.asc][srczipasc] | -[bintar]: https://www.apache.org/dyn/closer.lua/logging/log4j/${Log4jAuditVersion}/apache-log4j-audit-${Log4jAuditVersion}-bin.tar.gz -[bintarmd5]: https://www.apache.org/dist/logging/log4j/${Log4jAuditVersion}/apache-log4j-audit-${Log4jAuditVersion}-bin.tar.gz.md5 -[bintarasc]: https://www.apache.org/dist/logging/log4j/${Log4jAuditVersion}/apache-log4j-audit-${Log4jAuditVersion}-bin.tar.gz.asc -[binzip]: https://www.apache.org/dyn/closer.lua/logging/log4j/${Log4jAuditVersion}/apache-log4j-audit-${Log4jAuditVersion}-bin.zip -[binzipmd5]: https://www.apache.org/dist/logging/log4j/${Log4jAuditVersion}/apache-log4j-audit-${Log4jAuditVersion}-bin.zip.md5 -[binzipasc]: https://www.apache.org/dist/logging/log4j/${Log4jAuditVersion}/apache-log4j-audit-${Log4jAuditVersion}-bin.zip.asc -[srctar]: https://www.apache.org/dyn/closer.lua/logging/log4j/${Log4jAuditVersion}/apache-log4j-audit-${Log4jAuditVersion}-src.tar.gz -[srctarmd5]: https://www.apache.org/dist/logging/log4j/${Log4jAuditVersion}/apache-log4j-audit-${Log4jAuditVersion}-src.tar.gz.md5 -[srctarasc]: https://www.apache.org/dist/logging/log4j/${Log4jAuditVersion}/apache-log4j-audit-${Log4jAuditVersion}-src.tar.gz.asc -[srczip]: https://www.apache.org/dyn/closer.lua/logging/log4j/${Log4jAuditVersion}/apache-log4j-audit-${Log4jAuditVersion}-src.zip -[srczipmd5]: https://www.apache.org/dist/logging/log4j/${Log4jAuditVersion}/apache-log4j-audit-${Log4jAuditVersion}-src.zip.md5 -[srczipasc]: https://www.apache.org/dist/logging/log4j/${Log4jAuditVersion}/apache-log4j-audit-${Log4jAuditVersion}-src.zip.asc +[bintar]: https://www.apache.org/dyn/closer.lua/logging/log4j/log4j-audit/${Log4jAuditVersion}/apache-log4j-audit-${Log4jAuditVersion}-bin.tar.gz +[bintarmd5]: https://www.apache.org/dist/logging/log4j/log4j-audit/${Log4jAuditVersion}/apache-log4j-audit-${Log4jAuditVersion}-bin.tar.gz.md5 +[bintarasc]: https://www.apache.org/dist/logging/log4j/log4j-audit/${Log4jAuditVersion}/apache-log4j-audit-${Log4jAuditVersion}-bin.tar.gz.asc +[binzip]: https://www.apache.org/dyn/closer.lua/logging/log4j/log4j-audit/${Log4jAuditVersion}/apache-log4j-audit-${Log4jAuditVersion}-bin.zip +[binzipmd5]: https://www.apache.org/dist/logging/log4j/log4j-audit/${Log4jAuditVersion}/apache-log4j-audit-${Log4jAuditVersion}-bin.zip.md5 +[binzipasc]: https://www.apache.org/dist/logging/log4j/log4j-audit/${Log4jAuditVersion}/apache-log4j-audit-${Log4jAuditVersion}-bin.zip.asc +[srctar]: https://www.apache.org/dyn/closer.lua/logging/log4j/log4j-audit/${Log4jAuditVersion}/apache-log4j-audit-${Log4jAuditVersion}-src.tar.gz +[srctarmd5]: https://www.apache.org/dist/logging/log4j/log4j-audit/${Log4jAuditVersion}/apache-log4j-audit-${Log4jAuditVersion}-src.tar.gz.md5 +[srctarasc]: https://www.apache.org/dist/logging/log4j/log4j-audit/${Log4jAuditVersion}/apache-log4j-audit-${Log4jAuditVersion}-src.tar.gz.asc +[srczip]: https://www.apache.org/dyn/closer.lua/logging/log4j/log4j-audit/${Log4jAuditVersion}/apache-log4j-audit-${Log4jAuditVersion}-src.zip +[srczipmd5]: https://www.apache.org/dist/logging/log4j/log4j-aduit/${Log4jAuditVersion}/apache-log4j-audit-${Log4jAuditVersion}-src.zip.md5 +[srczipasc]: https://www.apache.org/dist/logging/log4j/log4j-audit/${Log4jAuditVersion}/apache-log4j-audit-${Log4jAuditVersion}-src.zip.asc It is essential that you verify the integrity of the downloaded files using the PGP or MD5 signatures. Please read [Verifying Apache HTTP Server Releases](https://httpd.apache.org/dev/verification.html) for more @@ -63,15 +63,6 @@ Apache Log4j ${Log4jAuditVersion} is signed by ${Log4jReleaseManager} (${Log4jRe Alternatively, you can verify the MD5 signature on the files. A unix program called md5 or md5sum is included in many unix distributions. -All previous releases of Apache log4j can be found in the -[archive repository](https://archive.apache.org/dist/logging/log4j/). +All previous releases of Apache log4j-Audit can be found in the +[archive repository](https://archive.apache.org/dist/logging/log4j/log4j-audit/). -$h2 Using Log4j on your classpath - -To use Log4j 2 in your application make sure that both the API and Core jars are in the application's classpath. Add -the dependencies listed below to your classpath. - - log4j-api-${Log4jAuditVersion}.jar - log4j-core-${Log4jAuditVersion}.jar - -You can do this from the command line or a manifest file. http://git-wip-us.apache.org/repos/asf/logging-log4j-audit/blob/af897a5b/src/site/markdown/index.md.vm ---------------------------------------------------------------------- diff --git a/src/site/markdown/index.md.vm b/src/site/markdown/index.md.vm index 5199f10..ea6470b 100644 --- a/src/site/markdown/index.md.vm +++ b/src/site/markdown/index.md.vm @@ -33,7 +33,7 @@ $h3 What is the difference between audit logging and "normal" logging? In a typical application developers add logging statements at key points to help diagnose problems or to document unexpected occurances, such as a failure to communicate with a key service. These are normally referred -to as diagnostic logs. Diagnostic logs are critical in aiding in maintaining the servicability of the application +to as diagnostic logs. Diagnostic logs are critical in maintaining the servicability of the application but generally aren't very useful in helping to determine who made a paticular change and when it was done. On the other hand, audit logs focus on identifying when a change was made, who made it, and what data elements were changed. While audit logs can sometimes help in troubleshooting problems, that isn't their primary purpose just as @@ -77,8 +77,18 @@ would. $h2 Features +Each application has its own events that need to be audited. Before using Log4j Audit, application owners need to define +the AuditMessages that capture the exact attributes of that need to be captured. The [Getting Started](link) page +provides a tutorial that explains how to define audit events for an application. + $h3 Audit Event Catalog +Once audit events are defined, they need to be maintained: as the application evolves, developers will inevitably +discover they need to add, remove or change attributes of the audit events. Log4j Audit can persist the audit event +definitions in a JSON file. This file becomes the Audit Event Catalog for the application. Log4j Audit is designed +to store the event definition file in a Git repository so that the evolution of the audit events themselves have an +audit trail in the Git history of the file. Log4j Audit provides a web interface for editing the events. + Log4j Audit uses a catalog of events to determine what events can be logged and to validate the events. Java interfaces are generated from the catalog for applications to use when generating the audit events. A web UI is provided to edit and manage the catalog but the UI requires a specific user's Git credentials and @@ -86,9 +96,15 @@ so is best run on the local user's computer. $h3 Dynamic Event Catalog +The typical event catalog is considered static. It is defined during the development process and cannot be changed in +a running system without a new release. + In addition to the catalog that is managed by the web UI a set of dynamic catalogs can also be used. These catalogs can be used by multi-tenant applications where each tenant has a different set of events and/or attributes that need -to be logged. +to be logged. The dynamic events are defined while the application is running and because they are specific to a +tenant in the running system. Because they are dynamic they are maintained in a database. Changes to the dynamic +catalog do not affect the static catalog. Changes to the dynamic catalog are performed using the audit service's REST +APi. $h3 Audit Service http://git-wip-us.apache.org/repos/asf/logging-log4j-audit/blob/af897a5b/src/site/markdown/requestContext.md.vm ---------------------------------------------------------------------- diff --git a/src/site/markdown/requestContext.md.vm b/src/site/markdown/requestContext.md.vm index 1fa5ad9..6d3d59d 100644 --- a/src/site/markdown/requestContext.md.vm +++ b/src/site/markdown/requestContext.md.vm @@ -25,7 +25,7 @@ Log4j-Audit incorporates Log4j ThreadContext variables as a way of including inf common throughout the application. While usage of a RequestContext is not required, the use of one should be considered a best practice. -In a typcial application that might use a ThreadContext Map, it will place variables in the Map so that they will +In a typical application that might use a ThreadContext Map, it will place variables in the Map so that they will be included in all Log4j log events. Log4j Audit takes advantage of this so that variables placed into the ThreadContext Map will be automatically be available in audit events as well. In addition, through the use of the RequestContextFilter and RequestContextHeaderInterceptor, these variable will automatically be passed from a @@ -37,7 +37,7 @@ can be correlated in the application and through all the microservices the appli include things like a client account number (especially useful in multi-tenant applications), the user's login id, and the ip address of the user. In a services-based application these values need to be populated at the point of entry to the application and then passed to all the service endpoints. It is especially useful to create artificial -request and session ids so that all the activities acrossall the servers can be correlated to the user's request +request and session ids so that all the activities across all the servers can be correlated to the user's request and/or session. When defining audit events there are generally two types of data included in the event. The first are data elements @@ -49,7 +49,7 @@ the data specific to that event needs to be added by the application. Another use case for RequestContext or ThreadContext Map variables is when lower level components need access to data elements that the components calling them have no use for. For example, an application may shard its database based on a customer's account number. A particular component may not need the account number in any of its operations -but would be forced to pass it through so the database access component could use it to determine which datbase to +but would be forced to pass it through so the database access component could use it to determine which database to use. By using the RequestContext the intermediate component no longer has to be aware of what customer it is performing work for. @@ -123,6 +123,10 @@ application (including audit logging). It is then converted to HTTP headers by a Log4j Audit) that are passed on a REST call to a service. The service then uses a servlet filter (provided by Log4j Audit) to create a RequestContext for use by the service. +This same mechanism can be applied to other kinds of distributed requests. For example, when using Spring AMQP the +RequestContext fields can be captured as AMQP headers using a MessagePostProcessor and the RequestContext recreated +in the consumer using an Interceptor. +  $h3 Initializing the RequestContext @@ -207,7 +211,7 @@ public class RequestContextInterceptor implements HandlerInterceptor { } ``` -$h3 Passing the RequestContext to a Service +$h3 Passing the RequestContext to a Spring REST Service Log4j Audit provides an Interceptor that can be used with Spring web applications that will convert RequestContext variables to HTTP headers. The RequestContextHeaderInterceptor can be created by a java configuration method such as: http://git-wip-us.apache.org/repos/asf/logging-log4j-audit/blob/af897a5b/src/site/site.xml ---------------------------------------------------------------------- diff --git a/src/site/site.xml b/src/site/site.xml index 4f1a6b8..1d1b971 100644 --- a/src/site/site.xml +++ b/src/site/site.xml @@ -31,8 +31,8 @@ <item name="About" href="/index.html"/> <item name="Download" href="/download.html"/> <item name="Getting Started" href="/gettingStarted.html"/> - <item name="RequestContext" href="/requestContext.html"/> <item name="Audit Catalog" href="catalog.html"/> + <item name="RequestContext" href="/requestContext.html"/> <item name="Sample Project" href="sample.html"/> <item name="Changelog" href="/changelog.html"/> <item name="Javadoc" href="/apidocs/index.html"/> http://git-wip-us.apache.org/repos/asf/logging-log4j-audit/blob/af897a5b/src/site/xdoc/gettingStarted.xml.vm ---------------------------------------------------------------------- diff --git a/src/site/xdoc/gettingStarted.xml.vm b/src/site/xdoc/gettingStarted.xml.vm index 525859c..6dc64af 100644 --- a/src/site/xdoc/gettingStarted.xml.vm +++ b/src/site/xdoc/gettingStarted.xml.vm @@ -78,7 +78,54 @@ <subsection name="Inspect the build results"> <p>List the contents of audit-service-api/target/generated-sources/log4j-audit directory. The event interfaces - generated from the catalog will be located in this directory.</p> + generated from the catalog will be located in this directory. As an example, the Class that represents + a transfer event looks like: + + <code><pre> + package org.apache.logging.log4j.audit.event; + + import java.math.BigDecimal; + import org.apache.logging.log4j.audit.AuditEvent; + import org.apache.logging.log4j.audit.annotation.Constraint; + import org.apache.logging.log4j.audit.annotation.MaxLength; + import org.apache.logging.log4j.audit.annotation.RequestContext; + import org.apache.logging.log4j.audit.annotation.Required; + + /** + * Transfer between accounts + * @author generated + */ + @MaxLength(32) + @RequestContext(key="hostName") + @RequestContext(key="loginId", required=true) + @RequestContext(key="ipAddress", constraints={@Constraint(constraintType="pattern", constraintValue="^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$")}) + @RequestContext(key="accountNumber", required=true) + @RequestContext(key="userId", required=true) + public interface Transfer extends AuditEvent { + + /** + * Amount : Amount to transfer + * @param amount Amount to transfer + */ + @Required + public void setAmount(BigDecimal amount); + + /** + * From Account Number : Source of funds + * @param fromAccount Source of funds + */ + @Required + public void setFromAccount(int fromAccount); + + /** + * To Account Number : Destination account + * @param toAccount Destination account + */ + @Required + @Constraint(constraintType="minValue", constraintValue="1") + public void setToAccount(int toAccount); + }</pre></code> + </p> </subsection> <a name="Run"/> <subsection name="Run an application that performs auditing"> @@ -91,6 +138,50 @@ ./sample-app.sh vi target/logs/audit.log</pre></code></li> </ol> + <p>The output from the logs should look similar to: + <code><pre><![CDATA[ + <128>1 2018-06-09T19:54:26.018-07:00 RalphGoers-MacBook-Pro.local SampleApp 18815 Audit [RequestContext@18060 hostName="RalphGoers-MacBook-Pro.local" ipAddress="192.168.1.15" loginId="testuser"][login@18060] + <128>1 2018-06-09T19:54:26.021-07:00 RalphGoers-MacBook-Pro.local SampleApp 18815 Audit [RequestContext@18060 accountNumber="12345" hostName="RalphGoers-MacBook-Pro.local" ipAddress="192.168.1.15" loginId="testuser" userId="1111"][login@18060 completionStatus="Success"] + <128>1 2018-06-09T19:54:26.026-07:00 RalphGoers-MacBook-Pro.local SampleApp 18815 Audit [RequestContext@18060 accountNumber="12345" hostName="RalphGoers-MacBook-Pro.local" ipAddress="192.168.1.15" loginId="testuser" userId="1111"][deposit@18060 account="123456" amount="100"] + <128>1 2018-06-09T19:54:26.027-07:00 RalphGoers-MacBook-Pro.local SampleApp 18815 Audit [RequestContext@18060 accountNumber="12345" hostName="RalphGoers-MacBook-Pro.local" ipAddress="192.168.1.15" loginId="testuser" userId="1111"][deposit@18060 account="123456" amount="100" completionStatus="Success"]]]></pre> + </code> + Note that the formatting is completely controlled by the Log4j configuration. In this case, the RFC5424Layout was used.</p> + + <p>The application that generated these logs is: + <code><pre> + public class SampleApp { + + public static void main(String[] args) throws Exception { + String hostName = NetUtils.getLocalHostname(); + RequestContext.setHostName(hostName); + String inetAddress = InetAddress.getLocalHost().getHostAddress(); + RequestContext.setIpAddress(inetAddress); + RequestContext.setLoginId("testuser"); + Login login = LogEventFactory.getEvent(Login.class); + login.logEvent(); + String result = login("testuser"); + login.setCompletionStatus(result); + login.logEvent(); + Deposit deposit = LogEventFactory.getEvent(Deposit.class); + deposit.setAccount(123456); + deposit.setAmount(new BigDecimal(100.00)); + deposit.logEvent(); + result = deposit(deposit); + deposit.setCompletionStatus(result); + deposit.logEvent(); + RequestContext.clear(); + } + + private static String login(String user) { + RequestContext.setUserId("1111"); + RequestContext.setAccountNumber(12345L); + return "Success"; + } + + private static String deposit(Deposit deposit) { + return "Success"; + }</pre></code> + </p> </subsection> <a name="DeployAuditService"/> <subsection name="Deploy the Audit Service WAR">
