[
https://issues.apache.org/jira/browse/IMPALA-14634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18048763#comment-18048763
]
ASF subversion and git services commented on IMPALA-14634:
----------------------------------------------------------
Commit d5da6ee2a5b570fdb99d9da5d74885c48ec5d539 in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=d5da6ee2a ]
IMPALA-14634: Organize custom cluster logs into subdirectories by test name
This changes the logic for setting the log directory for
custom cluster tests. It uses the test class name and (optionally)
the test method name to put log files into subdirectories.
For a test that restarts the cluster for each test method, this
would have a directory structure of
base_dir / test class / test method name /
For a test that starts the cluster at the class level and reuses
it across tests methods, this would be:
base_dir / test class /
This directory structure can still have logs from multiple tests
going to a single directory. For example, some tests are parameterized
so that a test method runs multiple times with different parameter
values. Those will share a directory. This should be rare and most
directories will have logs from a single test.
Some test methods restart the cluster within the test itself rather
than just at the beginning. This stores the test method so these
restarts continue to use the same directory. The code is often in
@classmethod functions, so this stores the test method name at the
class level and manipulates it through @classmethod functions.
This is awkward, but it works because the custom cluster tests are
single threaded.
This also adjusts the logic for determining the base directory
for custom cluster logs. tests/run-custom-cluster-tests.sh sets
LOG_DIR to $IMPALA_CUSTOM_CLUSTER_TEST_LOGS_DIR (i.e.
logs/custom_cluster_tests) and this continues to respect LOG_DIR.
However, in the dev environment using impala-py.test, LOG_DIR
is not set and the code currently defaults to /tmp. This changes it
to use $IMPALA_CUSTOM_CLUSTER_TEST_LOGS_DIR so that logs go to
logs/custom_cluster_tests rather than /tmp.
Testing:
- Ran locally and verified the output goes into directories
- Ran exhaustive custom cluster tests
Change-Id: I8a8402fed1584a99f91451a3976e7026d0deb834
Reviewed-on: http://gerrit.cloudera.org:8080/23796
Reviewed-by: Michael Smith <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Quanlong Huang <[email protected]>
> Organize log files for custom cluster tests into subdirectories by test class
> / method
> --------------------------------------------------------------------------------------
>
> Key: IMPALA-14634
> URL: https://issues.apache.org/jira/browse/IMPALA-14634
> Project: IMPALA
> Issue Type: Task
> Components: Infrastructure
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
>
> Custom cluster tests put all impala log files in logs/custom_cluster_test.
> When running many of the tests, it can be hard to tell which log files (and
> minidumps, etc) belong to each test. If we used separate log directories for
> each test, then it would be clear. For example, logs for
> TestFooClass.test_method() could go to
> logs/custom_cluster_test/TestFooClass/test_method/.
> The downside of this is that all the logic that limits the space usage of log
> files doesn't apply. Broadly, we want to hold on to those files anyway, so we
> set IMPALA_MAX_LOG_FILES=0. Even so, this will hold onto more, so we should
> double-check disk space usage.
> Separately, when running custom cluster tests in a development environment
> using impala-py.test, the logs usually go to /tmp. It would be better to
> detect IMPALA_CUSTOM_CLUSTER_TEST_LOGS_DIR and put them there.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
