Arun Sarin created HDDS-9113:
--------------------------------
Summary: [snapshot] Snapshot path contents listing is failing
intermittently
Key: HDDS-9113
URL: https://issues.apache.org/jira/browse/HDDS-9113
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Manager
Reporter: Arun Sarin
Snapshot path contents listing is failing intermittently
{code:java}
2023-08-01 19:36:50,541|INFO|MainThread|machine.py:188 -
run()||GUID=e574a938-a16e-4a1e-9c42-5786f286e727|RUNNING:
/opt/cloudera/parcels/CDH/bin/ozone sh snapshot create
o3://ozone1/vol-fiisp/buck-6j174 snap-2yg7s
2023-08-01 19:36:54,676|INFO|MainThread|machine.py:230 -
run()||GUID=e574a938-a16e-4a1e-9c42-5786f286e727|Exit Code: 0
2023-08-01 19:36:54,679|INFO|MainThread|machine.py:188 -
run()||GUID=7dd9a1cf-a3ca-4260-9eb2-13cdafeb2519|RUNNING: klist -k -t
/home/hrt_qa/hadoopqa/keytabs/hrt_qa.headless.keytab | grep -v HTTP
2023-08-01 19:36:54,698|INFO|MainThread|machine.py:230 -
run()||GUID=7dd9a1cf-a3ca-4260-9eb2-13cdafeb2519|Exit Code: 0
2023-08-01 19:36:54,699|INFO|MainThread|machine.py:2131 -
get_principal_from_user()|--- user principal is [email protected]
2023-08-01 19:36:54,700|INFO|MainThread|machine.py:188 -
run()||GUID=d0dd3259-5244-416c-afc0-adf6185698ad|RUNNING:
/opt/cloudera/parcels/CDH/bin/ozone fs -ls -R
ofs://ozone1/vol-fiisp/buck-6j174/.snapshot/snap-2yg7s
2023-08-01 19:37:02,974|INFO|MainThread|machine.py:203 -
run()||GUID=d0dd3259-5244-416c-afc0-adf6185698ad|ls:
`ofs://ozone1/vol-fiisp/buck-6j174/.snapshot/snap-2yg7s': No such file or
directory
2023-08-01 19:37:03,023|INFO|MainThread|machine.py:232 -
run()||GUID=d0dd3259-5244-416c-afc0-adf6185698ad|Exit Code: 1
2023-08-01 19:37:03,023|INFO|MainThread|machine.py:238 - run()|Command
/opt/cloudera/parcels/CDH/bin/ozone fs -ls -R
ofs://ozone1/vol-fiisp/buck-6j174/.snapshot/snap-2yg7s failed after 0 retries
2023-08-01 19:37:03,037|INFO|MainThread|conftest.py:272 -
pytest_runtest_makereport()|call: <CallInfo when='call' exception: Snapshot
path contents list failed
assert 1 == 0>{code}
Snapshot creation was successful [validated in om-audit.log]
{noformat}
2023-08-01 19:36:54,627 | INFO | OMAudit | [email protected] |
ip=10.64.62.46 | op=CREATE_SNAPSHOT {volume=vol-fiisp, bucket=buck-6j174,
snapshotName=snap-2yg7s} | ret=SUCCESS | {noformat}
The volume and bucket has space and namespace quota set -
{code:java}
Volume quota for current test SpaceQuota: 8053063680,NameSpace quota: 19
Bucket quota for current test SpaceQuota: 8053063680,NameSpace quota: 19 {code}
ozone-om.log error stack-trace :
{code:java}
2023-08-01 19:36:50,236 INFO
[SstFilteringService#0]-org.apache.hadoop.ozone.om.snapshot.SnapshotCache:
Loading snapshot. Table key: /vol-dsjsj/buck-h5ed5/snap-4w2ww
2023-08-01 19:36:50,237 ERROR
[SstFilteringService#0]-org.apache.hadoop.ozone.om.SstFilteringService: Error
during Snapshot sst filtering
FILE_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Unable to
load snapshot. Snapshot with table key '/vol-dsjsj/buck-h5ed5/snap-4w2ww' is no
longer active
at
org.apache.hadoop.ozone.om.snapshot.SnapshotCache.get(SnapshotCache.java:205)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotCache.get(SnapshotCache.java:151)
at
org.apache.hadoop.ozone.om.SstFilteringService$SstFilteringTask.call(SstFilteringService.java:178)
at
org.apache.hadoop.hdds.utils.BackgroundService$PeriodicalTask.lambda$run$0(BackgroundService.java:121)
at
java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
