[
https://issues.apache.org/jira/browse/SOLR-18303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SOLR-18303:
----------------------------------
Labels: pull-request-available (was: )
> Incremental backup failure-cleanup path masks the real error with an
> unrelated `NoSuchFileException`
> ----------------------------------------------------------------------------------------------------
>
> Key: SOLR-18303
> URL: https://issues.apache.org/jira/browse/SOLR-18303
> Project: Solr
> Issue Type: Bug
> Components: Backup/Restore
> Affects Versions: 10.0
> Reporter: Philipp Trulson
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Disclaimer: Bug report was created with Claude, but I proof-read it and made
> sure the issue is valid and well described.
> When an incremental collection backup ({{{}action=BACKUP{}}}) fails during
> the shard-copy phase, {{BackupCmd.call()}} logs the real exception and then
> attempts to clean up the partially-created backup via
> {{CollectionHandlingUtils.cleanBackup()}} →
> {{{}DeleteBackupCmd.deleteBackupIds(){}}}. That cleanup code tries to delete
> the backup's {{zk_backup_<id>}} directory defensively:
> {code:java}
> // DeleteBackupCmd.java:210-217 (10.0.0), :211-215 (main)
> try {
> for (BackupId backupId : backupIdsDeletes) {
> repository.deleteDirectory(
> repository.resolveDirectory(backupUri,
> BackupFilePaths.getZkStateDir(backupId)));
> }
> } catch (FileNotFoundException e) {
> // ignore this
> }
> {code}
> {{zk_backup_<id>}} is only created later in {{BackupCmd.call()}} (via
> {{{}backupMgr.createZkStateDir(){}}}, which runs *after* the shard-copy
> step). If the shard-copy step fails, that directory was never created, so the
> delete attempt above is expected to be a no-op — but
> {{LocalFileSystemRepository.deleteDirectory()}} (via commons-io's
> {{PathUtils.deleteDirectory()}} / NIO {{{}Files.walkFileTree(){}}}) throws
> {{{}java.nio.file.NoSuchFileException{}}}, *not*
> {{{}java.io.FileNotFoundException{}}}. These are unrelated exception types
> ({{{}NoSuchFileException extends FileSystemException extends IOException{}}};
> it does not extend {{{}FileNotFoundException{}}}), so the catch clause never
> fires.
> The result: the {{NoSuchFileException}} propagates out of
> {{{}cleanBackup(){}}}, which is itself uncaught in {{{}BackupCmd.call(){}}}'s
> catch block, so it replaces the original {{throw e;}} that would have
> surfaced the real {{{}SolrException{}}}. The client — and the server log —
> only ever see:
> {noformat}
> java.nio.file.NoSuchFileException: <location>/<name>/<collection>/zk_backup_0
> {noformat}
> with no indication of what actually failed. The real error is logged one line
> earlier via {{{}log.error("Error happened during incremental backup for
> collection: {}", ...){}}}, but is otherwise discarded from the API response
> and easy to miss in the logs.
> h3. Steps to Reproduce
> Minimal, reliable repro — forces the shard-copy phase to fail for any reason:
> # Fresh SolrCloud node, default {{solr.security.allow.paths}} (i.e. not set
> to {{*}} and not covering the target location).
> # Create a collection: {{citibike}} (or any collection with at least one
> document).
> # {{GET
> /solr/admin/collections?action=BACKUP&name=citibike&collection=citibike&location=/tmp/some-backup-dir}}
> where {{/tmp/some-backup-dir}} is outside the default allowed paths
> ({{{}SOLR_HOME{}}}/{{{}SOLR_DATA_HOME{}}}/{{{}coreRootDirectory{}}}).
> h3. Expected Result
> HTTP 500 with the real cause, e.g.:
> {quote}Path ... must be relative to SOLR_HOME, SOLR_DATA_HOME
> coreRootDirectory. Set system property 'solr.security.allow.paths' to add
> other allowed paths.
> {quote}
> h3. Actual Result
> HTTP 500 with {{msg: "<location>/citibike/citibike/zk_backup_0"}} (bare
> {{{}NoSuchFileException{}}}, no explanation). The real cause above only
> appears in the server log, one ERROR line earlier, and is easy to overlook
> since the visible top-level exception is unrelated.
> *Note:* any other failure during shard-copy reproduces the same masking —
> e.g. this is effectively what happened in SOLR-15673 / SOLR-15696 for a
> shard-split-related parsing bug; fixing that trigger did not fix the masking
> itself, which is still present verbatim on {{main}} today.
> h3. Suggested Fix
> Two options, not mutually exclusive:
> # *Minimal:* also catch {{java.nio.file.NoSuchFileException}} (or better,
> check {{repository.exists(...)}} before calling {{{}deleteDirectory{}}}) at
> {{{}DeleteBackupCmd.java:214{}}}.
> # *More robust:* in {{{}BackupCmd.call(){}}}'s catch block, wrap the
> {{cleanBackup()}} call in its own try/catch that logs-and-swallows *any*
> exception from best-effort cleanup, so a secondary failure during rollback
> can never suppress the original {{{}throw e;{}}}. This closes the whole class
> of bug rather than just this one exception-type mismatch, and would have also
> prevented the confusion in SOLR-15673/15696.
> h3. Related
> SOLR-15673, SOLR-15696 — same symptom ({{{}NoSuchFileException{}}} on
> {{{}zk_backup_N{}}}), different root trigger (shard-split {{ShardBackupId}}
> parsing), masking mechanism left unfixed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
