[
https://issues.apache.org/jira/browse/SOLR-18586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18132788#comment-18132788
]
Andrzej Bialecki commented on SOLR-18586:
-----------------------------------------
It's not surprising that this fails on filesystems that don't support hard
linking. In any case, running Solr on NFS has always been strongly discouraged,
with different NFS implementations being subtly broken in different ways.
Perhaps we should improve the documentation to explicitly discourage this
instead of providing a workaround?
Also, a silent fallback to copy() seems dangerous because it can easily fill up
the volume and produce a DoS.
> TestTlogReplica testKillLeader, testRecovery and testOnlyLeaderIndexes fail
> repeatedly in the 10.1.0 RC3 smoke tests
> --------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-18586
> URL: https://issues.apache.org/jira/browse/SOLR-18586
> Project: Solr
> Issue Type: Test
> Affects Versions: 10.1
> Reporter: Nick Shanin
> Priority: Major
>
> h2. What was seen
> In five completed runs of the Solr 10.1.0 RC3 smoke tester (artifacts rev
> {{749712b71940c9f96d824c533f32f5c56dcd3924}}, JDK 21.0.12.1, aarch64), the
> build failed every time. {{TestTlogReplica.testKillLeader}} and
> {{TestTlogReplica.testRecovery}} failed in all five runs and
> {{testOnlyLeaderIndexes}} in four of the five, across different random seeds
> (for example {{787C6229591C6F1E}}, {{8D8640B367EDAD08}}, {{5AB00AEB1AE2392}}).
> h2. Host finding
> Matched pairs use the same seed and host; only the test temp directory
> differs (NFS {{/workspace}} vs local {{/tmp}}):
> * {{testKillLeader}} and {{testRecovery}}: fail on NFS with {{Too many
> links}} (EMLINK) from the hard links in {{IndexFetcher.java:1184}}
> ({{Files.createLink}}), and pass on local disk at the same seed.
> * {{testOnlyLeaderIndexes}} at seed {{5EC47295A32A8342}} (from run 005):
> fails on NFS with "Some replicas are not in sync with leader" and 21 EMLINK
> errors, and passes on local disk at the same seed. At seed
> {{3635CC89D23D0929}} it passes on both, so the failure is seed-dependent.
> These are one run per side per seed, so this is strong evidence for a host
> cause rather than proof. No code change has been identified for these three
> methods. A clean local-disk smoke run of RC3 has not completed yet, because
> the host restarted during the runs.
> h2. Related RC3 smoke failures
> RecoveryAfterSoftCommitTest is SOLR-18530 (fix in PR #5098) and
> TestCoordinatorRole.testNRTRestart is SOLR-16630 (fix in PR #5106); neither
> fix is in RC3. testNRTRestart also fails on NFS with EMLINK and passes on
> local disk at seed {{787C6229591C6F1E}}. This ticket covers the TlogReplica
> methods.
> h2. Fix branch (not a PR yet)
> https://github.com/nick-boss-tech/solr/tree/emlink-copy-fallback (head
> {{23053d08816}})
> * [fdb785484c6|https://github.com/nick-boss-tech/solr/commit/fdb785484c6]:
> {{IndexFetcher}} falls back to {{Files.copy}} when {{Files.createLink}} fails
> (EMLINK on NFS).
> * {{23053d08816}}: JS client install uses {{--include=dev}} so the build does
> not depend on {{NODE_ENV}}. Unrelated to SOLR-18586.
> Tested on NFS at the seeds above: {{TestTlogReplica}},
> {{TestCoordinatorRole}} and the replication test group pass with zero EMLINK
> errors. The full RC3 smoke run on this head has not completed a clean gate
> yet (the final binary-release step has no result). No PR is open.
> 🤖 _AI text above_ 🤖 _(reviewed by Nick Shanin)_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
