[
https://issues.apache.org/jira/browse/HADOOP-13951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HADOOP-13951:
---------------------------------
Description:
Right now this is expressed as failed Precommit-YARN-build jobs when they run
on H5 / H6 (see INFRA-13148), but the problem exists for all of the hadoop
related precommit jobs.
The issue is that we have some tests in Common (and maybe HDFS) that
purposefully set permissions within the {{target/}} directory to simulate a
failure to interact with underlying fs data. The test sets some subdirectories
to have permissions such that we can no longer delete their contents.
Right now our precommit jobs include a step post-yetus-test-patch that
traverses the target directories and ensures that all subdirectories are
modifiable:
{code}
find ${WORKSPACE} -name target | xargs chmod -R u+w
{code}
Unfortunately, if we don't get to that line (say due to an aborted build, or if
the call to yetus test-patch exceeds the job timeout), then we are left in a
state where there are still subdirectories that can't be modified (including
deleted).
Our builds also currently attempt to run a {{git clean}} at the very start of
the build after the repo is updated. If we have one of the aforementioned
timeouts that leaves a can't-be-deleted test directory, then all future builds
on that machine will fail attempting to run the {{git clean}} command.
was:
Right now this is expressed as failed Precommit-YARN-build jobs when they run
on H5 / H6 (see INFRA-13148), but the problem exists for all of the hadoop
related precommit jobs.
The issue is that we have some tests in Common (and maybe HDFS) that
purposefully set permissions within the {{target/}} directory to simulate a
failure to interact with underlying fs data. The test sets some subdirectories
to have permissions such that we can no longer delete their contents.
Right now our precommit job include a step post-yetus-test-patch that traverses
the target directories and ensure that all subdirectories are modifiable:
{code}
find ${WORKSPACE} -name target | xargs chmod -R u+w
{code}
Unfortunately, if we don't get to that line (say due to an aborted build, or if
the call to yetus test-patch exceeds the job timeout), then we are left in a
state where there are still subdirectories that can't be modified (including
deleted).
Our builds also currently attempt to run a {{git clean}} at the very start of
the build after the repo is updated. If we have one of the aforementioned
timeouts that leaves a can't-be-deleted test directory, then all future builds
on that machine will fail attempting to run the {{git clean}} command.
> Precommit builds do not adequately protect against test malformed fs
> permissions.
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-13951
> URL: https://issues.apache.org/jira/browse/HADOOP-13951
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
> Reporter: Sean Busbey
> Priority: Critical
>
> Right now this is expressed as failed Precommit-YARN-build jobs when they run
> on H5 / H6 (see INFRA-13148), but the problem exists for all of the hadoop
> related precommit jobs.
> The issue is that we have some tests in Common (and maybe HDFS) that
> purposefully set permissions within the {{target/}} directory to simulate a
> failure to interact with underlying fs data. The test sets some
> subdirectories to have permissions such that we can no longer delete their
> contents.
> Right now our precommit jobs include a step post-yetus-test-patch that
> traverses the target directories and ensures that all subdirectories are
> modifiable:
> {code}
> find ${WORKSPACE} -name target | xargs chmod -R u+w
> {code}
> Unfortunately, if we don't get to that line (say due to an aborted build, or
> if the call to yetus test-patch exceeds the job timeout), then we are left in
> a state where there are still subdirectories that can't be modified
> (including deleted).
> Our builds also currently attempt to run a {{git clean}} at the very start of
> the build after the repo is updated. If we have one of the aforementioned
> timeouts that leaves a can't-be-deleted test directory, then all future
> builds on that machine will fail attempting to run the {{git clean}} command.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
