[
https://issues.apache.org/jira/browse/HDDS-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy updated HDDS-17857:
----------------------------------
Description:
h3. Problem
Several tests turn on audit logging by setting
\{{log4j.configurationFile=auditlog.properties}}. Every
\{{auditlog.properties}} under \{{src/test/resources}} names its file with a
relative path:
{noformat}
appender.audit.fileName=audit.log
appender.systemaudit.fileName=system_audit.log (integration-test only)
{noformat}
Log4j resolves these paths against the working directory of the surefire JVM,
which is the module directory, so the files land next to \{{pom.xml}} and
\{{src/}}, not under \{{target/}}. They show up as untracked files in \{{git
status}} (\{{.gitignore}} doesn't cover them), and \{{mvn clean}} doesn't
remove them, so they pile up between runs. A careless \{{git add -A}} can
commit them.
Example: after running \{{TestSecureOzoneRpcClient}}, \{{git status}} shows
{noformat}
?? hadoop-ozone/integration-test/audit.log
?? hadoop-ozone/integration-test/system_audit.log
{noformat}
{\{OzoneRpcClientTests}} (the base of \{{TestOzoneRpcClient}} and
\{{TestSecureOzoneRpcClient}}) calls \{{AuditLogTestUtils.enableAuditLog()}}
but never deletes the files. Other tests clean up only partly: \{{TestOmAcls}}
and \{{TestOMRatisSnapshotTransfer}} delete them, and \{{TestOzoneAuditLogger}}
deletes \{{audit.log}} in \{{@AfterAll}}. That cleanup doesn't run if the fork
is killed or times out.
Affected configs (only the integration-test one also writes
\{{system_audit.log}}):
{noformat}
hadoop-ozone/integration-test/src/test/resources/auditlog.properties
hadoop-hdds/framework/src/test/resources/auditlog.properties
hadoop-hdds/rocks-native/src/test/resources/auditlog.properties
hadoop-ozone/s3gateway/src/test/resources/auditlog.properties
{noformat}
The readers use the same relative names: \{{AuditLogTestUtils}}
(\{{AUDITLOG_FILENAME}}, \{{SYSTEM_AUDITLOG_FILENAME}}),
\{{TestOzoneAuditLogger}}, \{{TestS3GatewayAuditLog}} and
\{{TestOzoneRpcClientForAclAuditLog}}.
h3. Proposed fix
Write the files under the module's \{{target/}} directory. Surefire already
sets the system property \{{hadoop.log.dir}} to the module's \{{target/log}}
directory in every test JVM (the \{{systemPropertyVariables}} of the surefire
plugin in the root \{{pom.xml}}), so:
# In each test \{{auditlog.properties}}, use a log4j2 system property lookup
with a fallback for IDE runs, e.g.
{noformat}
appender.audit.fileName=${sys:hadoop.log.dir:-target/log}/audit.log
appender.systemaudit.fileName=${sys:hadoop.log.dir:-target/log}/system_audit.log
{noformat}
# Resolve the same location in \{{AuditLogTestUtils}} (e.g.
\{{Paths.get(System.getProperty("hadoop.log.dir", "target/log"),
"audit.log")}}) and use it in the tests that open the file directly
(\{{TestOzoneAuditLogger}}, \{{TestS3GatewayAuditLog}},
\{{TestOzoneRpcClientForAclAuditLog}}), so that writers and readers agree.
# Optionally, add \{{audit.log}} and \{{system_audit.log}} to \{{.gitignore}},
so that files left by older runs or branches don't show up.
h3. Compatibility / side effects
Test-only. The production audit configs in
\{{hadoop-ozone/dist/src/shell/conf/*-audit-log4j2.properties}} don't change.
\{{mvn clean}} will now remove the files.
h3. Out of scope
Tests that run in parallel forks of the same module still share one file,
because they share one \{{target/log}}. Giving each fork its own directory
(e.g. with the \{{surefire.forkNumber}} property) would be a separate change.
h3. Tests
* Run the audit tests listed above and check that they still pass.
* After the run, \{{git status}} shows no \{{audit.log}} or
\{{system_audit.log}}, and the files are under \{{<module>/target/log/}}.
was:
h3. Problem
Several tests turn on audit logging by setting
\{{log4j.configurationFile=auditlog.properties}}. Every
\{{auditlog.properties}} under \{{src/test/resources}} names its file with a
relative path:
{noformat}
appender.audit.fileName=audit.log
appender.systemaudit.fileName=system_audit.log (integration-test only)
{noformat}
Log4j resolves these paths against the working directory of the surefire JVM,
which is the module directory, so the files land next to \{{pom.xml}} and
\{{src/}}, not under \{{target/}}. They show up as untracked files in \{{git
status}} (\{{.gitignore}} doesn't cover them), and \{{mvn clean}} doesn't
remove them, so they pile up between runs. A careless \{{git add -A}} can
commit them.
Example: after running \{{TestSecureOzoneRpcClient}}, \{{git status}} shows
{noformat}
?? hadoop-ozone/integration-test/audit.log
?? hadoop-ozone/integration-test/system_audit.log
{noformat}
{\{OzoneRpcClientTests}} (the base of \{{TestOzoneRpcClient}} and
\{{TestSecureOzoneRpcClient}}) calls \{{AuditLogTestUtils.enableAuditLog()}}
but never deletes the files. Other tests clean up only partly: \{{TestOmAcls}}
and \{{TestOMRatisSnapshotTransfer}} delete them, and \{{TestOzoneAuditLogger}}
deletes \{{audit.log}} in \{{@AfterAll}}. That cleanup doesn't run if the fork
is killed or times out.
Affected configs:
* \{{hadoop-ozone/integration-test/src/test/resources/auditlog.properties}}
(\{{audit.log}}, \{{system_audit.log}})
* \{{hadoop-hdds/framework/src/test/resources/auditlog.properties}}
* \{{hadoop-hdds/rocks-native/src/test/resources/auditlog.properties}}
* \{{hadoop-ozone/s3gateway/src/test/resources/auditlog.properties}}
The readers use the same relative names: \{{AuditLogTestUtils}}
(\{{AUDITLOG_FILENAME}}, \{{SYSTEM_AUDITLOG_FILENAME}}),
\{{TestOzoneAuditLogger}}, \{{TestS3GatewayAuditLog}} and
\{{TestOzoneRpcClientForAclAuditLog}}.
h3. Proposed fix
Write the files under the module's \{{target/}} directory. Surefire already
passes \{{hadoop.log.dir=$\{project.build.directory\}/log}} to every test JVM
(root \{{pom.xml}}, \{{systemPropertyVariables}}), so:
# In each test \{{auditlog.properties}}, use a log4j2 system property lookup
with a fallback for IDE runs, e.g.
{noformat}
appender.audit.fileName=${sys:hadoop.log.dir:-target/log}/audit.log
appender.systemaudit.fileName=${sys:hadoop.log.dir:-target/log}/system_audit.log
{noformat}
# Resolve the same location in \{{AuditLogTestUtils}} (e.g.
\{{Paths.get(System.getProperty("hadoop.log.dir", "target/log"),
"audit.log")}}) and use it in the tests that open the file directly
(\{{TestOzoneAuditLogger}}, \{{TestS3GatewayAuditLog}},
\{{TestOzoneRpcClientForAclAuditLog}}), so that writers and readers agree.
# Optionally, add \{{audit.log}} and \{{system_audit.log}} to \{{.gitignore}},
so that files left by older runs or branches don't show up.
h3. Compatibility / side effects
Test-only. The production audit configs in
\{{hadoop-ozone/dist/src/shell/conf/*-audit-log4j2.properties}} don't change.
\{{mvn clean}} will now remove the files.
h3. Out of scope
Tests that run in parallel forks of the same module still share one file,
because they share one \{{target/log}}. Separating them (e.g. with
\{{$\{surefire.forkNumber\}}}) would be a separate change.
h3. Tests
* Run the audit tests listed above and check that they still pass.
* After the run, \{{git status}} shows no \{{audit.log}} or
\{{system_audit.log}}, and the files are under \{{<module>/target/log/}}.
> Tests write audit.log and system_audit.log into the module directory instead
> of target
> --------------------------------------------------------------------------------------
>
> Key: HDDS-17857
> URL: https://issues.apache.org/jira/browse/HDDS-17857
> Project: Apache Ozone
> Issue Type: Bug
> Components: test
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> h3. Problem
> Several tests turn on audit logging by setting
> \{{log4j.configurationFile=auditlog.properties}}. Every
> \{{auditlog.properties}} under \{{src/test/resources}} names its file with a
> relative path:
> {noformat}
> appender.audit.fileName=audit.log
> appender.systemaudit.fileName=system_audit.log (integration-test only)
> {noformat}
> Log4j resolves these paths against the working directory of the surefire JVM,
> which is the module directory, so the files land next to \{{pom.xml}} and
> \{{src/}}, not under \{{target/}}. They show up as untracked files in \{{git
> status}} (\{{.gitignore}} doesn't cover them), and \{{mvn clean}} doesn't
> remove them, so they pile up between runs. A careless \{{git add -A}} can
> commit them.
> Example: after running \{{TestSecureOzoneRpcClient}}, \{{git status}} shows
> {noformat}
> ?? hadoop-ozone/integration-test/audit.log
> ?? hadoop-ozone/integration-test/system_audit.log
> {noformat}
> {\{OzoneRpcClientTests}} (the base of \{{TestOzoneRpcClient}} and
> \{{TestSecureOzoneRpcClient}}) calls \{{AuditLogTestUtils.enableAuditLog()}}
> but never deletes the files. Other tests clean up only partly:
> \{{TestOmAcls}} and \{{TestOMRatisSnapshotTransfer}} delete them, and
> \{{TestOzoneAuditLogger}} deletes \{{audit.log}} in \{{@AfterAll}}. That
> cleanup doesn't run if the fork is killed or times out.
> Affected configs (only the integration-test one also writes
> \{{system_audit.log}}):
> {noformat}
> hadoop-ozone/integration-test/src/test/resources/auditlog.properties
> hadoop-hdds/framework/src/test/resources/auditlog.properties
> hadoop-hdds/rocks-native/src/test/resources/auditlog.properties
> hadoop-ozone/s3gateway/src/test/resources/auditlog.properties
> {noformat}
> The readers use the same relative names: \{{AuditLogTestUtils}}
> (\{{AUDITLOG_FILENAME}}, \{{SYSTEM_AUDITLOG_FILENAME}}),
> \{{TestOzoneAuditLogger}}, \{{TestS3GatewayAuditLog}} and
> \{{TestOzoneRpcClientForAclAuditLog}}.
> h3. Proposed fix
> Write the files under the module's \{{target/}} directory. Surefire already
> sets the system property \{{hadoop.log.dir}} to the module's \{{target/log}}
> directory in every test JVM (the \{{systemPropertyVariables}} of the surefire
> plugin in the root \{{pom.xml}}), so:
> # In each test \{{auditlog.properties}}, use a log4j2 system property lookup
> with a fallback for IDE runs, e.g.
> {noformat}
> appender.audit.fileName=${sys:hadoop.log.dir:-target/log}/audit.log
> appender.systemaudit.fileName=${sys:hadoop.log.dir:-target/log}/system_audit.log
> {noformat}
> # Resolve the same location in \{{AuditLogTestUtils}} (e.g.
> \{{Paths.get(System.getProperty("hadoop.log.dir", "target/log"),
> "audit.log")}}) and use it in the tests that open the file directly
> (\{{TestOzoneAuditLogger}}, \{{TestS3GatewayAuditLog}},
> \{{TestOzoneRpcClientForAclAuditLog}}), so that writers and readers agree.
> # Optionally, add \{{audit.log}} and \{{system_audit.log}} to
> \{{.gitignore}}, so that files left by older runs or branches don't show up.
> h3. Compatibility / side effects
> Test-only. The production audit configs in
> \{{hadoop-ozone/dist/src/shell/conf/*-audit-log4j2.properties}} don't change.
> \{{mvn clean}} will now remove the files.
> h3. Out of scope
> Tests that run in parallel forks of the same module still share one file,
> because they share one \{{target/log}}. Giving each fork its own directory
> (e.g. with the \{{surefire.forkNumber}} property) would be a separate change.
> h3. Tests
> * Run the audit tests listed above and check that they still pass.
> * After the run, \{{git status}} shows no \{{audit.log}} or
> \{{system_audit.log}}, and the files are under \{{<module>/target/log/}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]