[
https://issues.apache.org/jira/browse/SOLR-18586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18133116#comment-18133116
]
Nick Shanin commented on SOLR-18586:
------------------------------------
Thank you. On the silent copy fallback, I agree: silently copying a large index
during replication can fill a volume, and I am withdrawing that as the proposal.
One precision on the failure itself: it is not a filesystem without hard-link
support. The error is EMLINK, a link-count limit, thrown from the
{{createLink}} call in {{IndexFetcher}} during recovery replication. Hard links
work on this mount; the test runs create enough of them against the same index
files to hit the server's limit.
On NFS generally: agreed that Solr on NFS is discouraged for production, and
clearer documentation is fine with me. These are test runs, though (the RC3
smoke tests, on a host whose workspace sits on NFS). On that host the rest of
the suite passes, roughly 5,900 solr:core tests per run, and the repeating
failures concentrate in the three TestTlogReplica methods that drive the
linking, which is why I read this as link-count pressure in this code path
rather than general NFS instability.
If a production fallback is not wanted, I have taken the test side instead, on
a branch:
[solr-18586-tlog-nolink-submit|https://github.com/nick-boss-tech/solr/tree/solr-18586-tlog-nolink-submit],
commit
[3ea9ea74b4|https://github.com/nick-boss-tech/solr/commit/3ea9ea74b47fc8927d834edef7c05c769df7bfd9].
{{IndexFetcher}} gains an opt-in system property
({{solr.indexfetcher.disableHardLinks}}, off by default) that skips the hard
link reuse so identical files are downloaded, and TestTlogReplica sets it,
since that class exercises tlog recovery rather than the link optimization.
Verified by making {{createLink}} throw unconditionally in a scratch tree:
testRecovery passes with the property set and fails without it. Production
behavior is unchanged unless an operator opts in, and nothing falls back
silently. Is this shape acceptable, or would you rather the ticket close as a
documented NFS limitation?
🤖 _AI text above_ 🤖 _(reviewed by Nick Shanin)_
> TestTlogReplica testKillLeader, testRecovery and testOnlyLeaderIndexes fail
> repeatedly in the 10.1.0 RC3 smoke tests
> --------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-18586
> URL: https://issues.apache.org/jira/browse/SOLR-18586
> Project: Solr
> Issue Type: Test
> Affects Versions: 10.1
> Reporter: Nick Shanin
> Priority: Minor
>
> h2. What was seen
> In five completed runs of the Solr 10.1.0 RC3 smoke tester (artifacts rev
> {{749712b71940c9f96d824c533f32f5c56dcd3924}}, JDK 21.0.12.1, aarch64), the
> build failed every time. {{TestTlogReplica.testKillLeader}} and
> {{TestTlogReplica.testRecovery}} failed in all five runs and
> {{testOnlyLeaderIndexes}} in four of the five, across different random seeds
> (for example {{787C6229591C6F1E}}, {{8D8640B367EDAD08}}, {{5AB00AEB1AE2392}}).
> h2. Host finding
> Matched pairs use the same seed and host; only the test temp directory
> differs (NFS {{/workspace}} vs local {{/tmp}}):
> * {{testKillLeader}} and {{testRecovery}}: fail on NFS with {{Too many
> links}} (EMLINK) from the hard links in {{IndexFetcher.java:1184}}
> ({{Files.createLink}}), and pass on local disk at the same seed.
> * {{testOnlyLeaderIndexes}} at seed {{5EC47295A32A8342}} (from run 005):
> fails on NFS with "Some replicas are not in sync with leader" and 21 EMLINK
> errors, and passes on local disk at the same seed. At seed
> {{3635CC89D23D0929}} it passes on both, so the failure is seed-dependent.
> These are one run per side per seed, so this is strong evidence for a host
> cause rather than proof. No code change has been identified for these three
> methods. A clean local-disk smoke run of RC3 has not completed yet, because
> the host restarted during the runs.
> h2. Related RC3 smoke failures
> RecoveryAfterSoftCommitTest is SOLR-18530 (fix in PR #5098) and
> TestCoordinatorRole.testNRTRestart is SOLR-16630 (fix in PR #5106); neither
> fix is in RC3. testNRTRestart also fails on NFS with EMLINK and passes on
> local disk at seed {{787C6229591C6F1E}}. This ticket covers the TlogReplica
> methods.
> h2. Fix branch (not a PR yet)
> https://github.com/nick-boss-tech/solr/tree/emlink-copy-fallback (head
> {{23053d08816}})
> * [fdb785484c6|https://github.com/nick-boss-tech/solr/commit/fdb785484c6]:
> {{IndexFetcher}} falls back to {{Files.copy}} when {{Files.createLink}} fails
> (EMLINK on NFS).
> * {{23053d08816}}: JS client install uses {{--include=dev}} so the build does
> not depend on {{NODE_ENV}}. Unrelated to SOLR-18586.
> Tested on NFS at the seeds above: {{TestTlogReplica}},
> {{TestCoordinatorRole}} and the replication test group pass with zero EMLINK
> errors. The full RC3 smoke run on this head has not completed a clean gate
> yet (the final binary-release step has no result). No PR is open.
> 🤖 _AI text above_ 🤖 _(reviewed by Nick Shanin)_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
