Brian, please find my replies inline below.
In summary, your advice helped, and I was able to delete some repos. I
have documented the solution in detail below so that any others afflicted
by this problem may find it.
> use pulp-admin to list tasks: `pulp-admin tasks list`. The list is
> long and the ordering isn't as you probably would expect so make sure
> to look at it all. You should look for tasks in the running state and
> manually cancel them by uuid with `pulp-admin tasks cancel --task-id`.
This is a little impractical when there are ~550 tasks... but the
following bash fragment does it.
----------------------------------------------------------------------
function CancelAllRunningTasks() {
local AllTaskIds=( $(pulp-admin tasks list | perl -0777 -pe
's/Operations:[^\n]*\nResources:[^\n]*\nState: +Running\nStart Time: +[^
\n]+\nFinish Time: +[^ \n]+\nTask Id: +([^ \n]+)/\1/igs' | grep -e
"^[a-f0-9\-]\+\$") )
local i
for ((i=0; i<${#AllTaskIds[@]}; ++i ))
do
pulp-admin tasks cancel --task-id ${AllTaskIds[i]}
done
}
----------------------------------------------------------------------
I have found that I have to run this several times, as pulp continues to
start new tasks.
I also have a sync schedule for every repo, so I thought I should remove
those too:
----------------------------------------------------------------------
function RemoveAllSyncSchedules() {
local RepoTypes="rpm iso"
local RepoType
for RepoType in $RepoTypes
do
local AllRepoNames=( $(pulp-admin $RepoType repo list
--fields=Id | sed
--quiet -e "s/Id: \+\([^ \t]\+\)/\1/p") )
local i
for (( i=0; i<${#AllRepoNames[@]}; ++i ))
do
local SCHEDULES=( $( pulp-admin $RepoType repo sync
schedules list
--repo-id=${AllRepoNames[i]} | sed --quiet -e "s/Id: \+\([^
\t]\+\)/\1/p") )
local j
for (( j=0; j<${#SCHEDULES[@]}; ++j ))
do
pulp-admin $RepoType repo sync schedules delete
--repo-id=${AllRepoNames[i]} --schedule-id=${SCHEDULES[j]}
done
done
done
}
----------------------------------------------------------------------
Even after running this, pulp is still running syncs. When the running
task list is emptied, I have continued running CancelAllRunningTasks, and
it continues to find running tasks to cancel, even after all this. Pulp
seems to be generating its own tasks (or perhaps Waiting tasks are being
changed to Running).
> Also gratuitously restarting pulp_workers and pulp_resource_manager
> may help as it does similar cancellations.
I have stopped and re-started pulp using the methods previously described.
The logs still show 219 tasks received on re-starting pulp.
So now I am trying this:
----------------------------------------------------------------------
while true; do sleep 10; date; CancelAllRunningTasks; done
----------------------------------------------------------------------
When this stops killing tasks, I re-start pulp.
Continuing in this fashion, the number of tasks received when pulp starts
up seems to be declining at each iteration:
219
81
32
3
> I expect after either these actions, new pulp tasks should execute
> correctly.
I am now able to delete repos. I have now cleared out atrpms, down to 343
repos.
I guess that demonstrates that new pulp tasks are executing correctly.
I am hoping to be able to sync some critical repos and get pulp upgraded
to 2.6, and put this terrible episode with pulp 2.5.1 behind me.
Thanks for the help,
Ben.
_______________________________________________
Pulp-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/pulp-list
