[
https://issues.apache.org/jira/browse/SOLR-18586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18132267#comment-18132267
]
Nick Shanin edited comment on SOLR-18586 at 10/27/26, 7:45 AM:
---------------------------------------------------------------
All of these are related to hard-links on NFS. {{testKillLeader}} and
{{testRecovery}} fail with the link errors directly; {{testOnlyLeaderIndexes}}
fails downstream of them, when recovery replication cannot complete, and it
passes on NFS once the link call is replaced by a copy.
🤖 _AI text above_ 🤖 _(reviewed by Nick Shanin)_
was (Author: JIRAUSER314749):
Seems like some but not all of these are related to hard-links on NFS.
> TestTlogReplica testKillLeader, testRecovery and testOnlyLeaderIndexes fail
> repeatedly in the 10.1.0 RC3 smoke tests
> --------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-18586
> URL: https://issues.apache.org/jira/browse/SOLR-18586
> Project: Solr
> Issue Type: Test
> Reporter: Nick Shanin
> Priority: Major
>
> h2. What was seen
> In five completed runs of the Solr 10.1.0 RC3 smoke tester (artifacts rev
> {{749712b71940c9f96d824c533f32f5c56dcd3924}}, JDK 21.0.12.1, aarch64), the
> build failed every time. {{TestTlogReplica.testKillLeader}} and
> {{TestTlogReplica.testRecovery}} failed in all five runs and
> {{testOnlyLeaderIndexes}} in four of the five, across different random seeds
> (for example {{787C6229591C6F1E}}, {{8D8640B367EDAD08}}, {{5AB00AEB1AE2392}}).
> h2. Host finding
> Matched pairs use the same seed and host; only the test temp directory
> differs (NFS {{/workspace}} vs local {{/tmp}}):
> * {{testKillLeader}} and {{testRecovery}}: fail on NFS with {{Too many
> links}} (EMLINK) from the hard links in {{IndexFetcher.java:1184}}
> ({{Files.createLink}}), and pass on local disk at the same seed.
> * {{testOnlyLeaderIndexes}} at seed {{5EC47295A32A8342}} (from run 005):
> fails on NFS with "Some replicas are not in sync with leader" and 21 EMLINK
> errors, and passes on local disk at the same seed. At seed
> {{3635CC89D23D0929}} it passes on both, so the failure is seed-dependent.
> These are one run per side per seed, so this is strong evidence for a host
> cause rather than proof. No code change has been identified for these three
> methods. A clean local-disk smoke run of RC3 has not completed yet, because
> the host restarted during the runs.
> h2. Related RC3 smoke failures
> RecoveryAfterSoftCommitTest is SOLR-18530 (fix in PR #5098) and
> TestCoordinatorRole.testNRTRestart is SOLR-16630 (fix in PR #5106); neither
> fix is in RC3. testNRTRestart also fails on NFS with EMLINK and passes on
> local disk at seed {{787C6229591C6F1E}}. This ticket covers the TlogReplica
> methods.
> h2. Fix branch (not a PR yet)
> https://github.com/nick-boss-tech/solr/tree/emlink-copy-fallback (head
> {{23053d08816}})
> * [fdb785484c6|https://github.com/nick-boss-tech/solr/commit/fdb785484c6]:
> {{IndexFetcher}} falls back to {{Files.copy}} when {{Files.createLink}} fails
> (EMLINK on NFS).
> * {{23053d08816}}: JS client install uses {{--include=dev}} so the build does
> not depend on {{NODE_ENV}}. Unrelated to SOLR-18586.
> Tested on NFS at the seeds above: {{TestTlogReplica}},
> {{TestCoordinatorRole}} and the replication test group pass with zero EMLINK
> errors. The full RC3 smoke run on this head has not completed a clean gate
> yet (the final binary-release step has no result). No PR is open.
> 🤖 _AI text above_ 🤖 _(reviewed by Nick Shanin)_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
