[
https://issues.apache.org/jira/browse/SOLR-18586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Shanin updated SOLR-18586:
-------------------------------
Description:
h2. What was seen
In five completed runs of the Solr 10.1.0 RC3 smoke tester (artifacts rev
{{749712b71940c9f96d824c533f32f5c56dcd3924}}, JDK 21.0.12.1, aarch64), the
build failed every time. {{TestTlogReplica.testKillLeader}} and
{{TestTlogReplica.testRecovery}} failed in all five runs and
{{testOnlyLeaderIndexes}} in four of the five, across different random seeds
(for example {{787C6229591C6F1E}}, {{8D8640B367EDAD08}}, {{5AB00AEB1AE2392}}).
h2. Host finding
Matched pairs use the same seed and host; only the test temp directory differs
(NFS {{/workspace}} vs local {{/tmp}}):
* {{testKillLeader}} and {{testRecovery}}: fail on NFS with {{Too many links}}
(EMLINK) from the hard links in {{IndexFetcher.java:1184}}
({{Files.createLink}}), and pass on local disk at the same seed.
* {{testOnlyLeaderIndexes}} at seed {{5EC47295A32A8342}} (from run 005): fails
on NFS with "Some replicas are not in sync with leader" and 21 EMLINK errors,
and passes on local disk at the same seed. At seed {{3635CC89D23D0929}} it
passes on both, so the failure is seed-dependent.
These are one run per side per seed, so this is strong evidence for a host
cause rather than proof. No code change has been identified for these three
methods. A clean local-disk smoke run of RC3 has not completed yet, because the
host restarted during the runs.
h2. Related RC3 smoke failures
RecoveryAfterSoftCommitTest is SOLR-18530 (fix in PR #5098) and
TestCoordinatorRole.testNRTRestart is SOLR-16630 (fix in PR #5106); neither fix
is in RC3. testNRTRestart also fails on NFS with EMLINK and passes on local
disk at seed {{787C6229591C6F1E}}. This ticket covers the TlogReplica methods.
h2. Fix branch (not a PR yet)
https://github.com/nick-boss-tech/solr/tree/emlink-copy-fallback (head
{{23053d08816}})
* {{fdb785484c6}}: {{IndexFetcher}} falls back to {{Files.copy}} when
{{Files.createLink}} fails (EMLINK on NFS).
* {{23053d08816}}: JS client install uses {{--include=dev}} so the build does
not depend on {{NODE_ENV}}. Unrelated to SOLR-18586.
Tested on NFS at the seeds above: {{TestTlogReplica}}, {{TestCoordinatorRole}}
and the replication test group pass with zero EMLINK errors. The full RC3 smoke
run on this head has not completed a clean gate yet (the final binary-release
step has no result). No PR is open.
🤖 _AI text above_ 🤖 _(reviewed by Nick Shanin)_
was:
h2. What was seen
In five completed runs of the Solr 10.1.0 RC3 smoke tester (artifacts rev
{{749712b71940c9f96d824c533f32f5c56dcd3924}}, JDK 21.0.12.1, aarch64), the
build failed every time. {{TestTlogReplica.testKillLeader}} and
{{TestTlogReplica.testRecovery}} failed in all five runs and
{{testOnlyLeaderIndexes}} in four of the five, across different random seeds
(for example {{787C6229591C6F1E}}, {{8D8640B367EDAD08}}, {{5AB00AEB1AE2392}}).
h2. Host finding
Matched pairs use the same seed and host; only the test temp directory differs
(NFS {{/workspace}} vs local {{/tmp}}):
* {{testKillLeader}} and {{testRecovery}}: fail on NFS with {{Too many links}}
(EMLINK) from the hard links in {{IndexFetcher.java:1184}}
({{Files.createLink}}), and pass on local disk at the same seed.
* {{testOnlyLeaderIndexes}} at seed {{5EC47295A32A8342}} (from run 005): fails
on NFS with "Some replicas are not in sync with leader" and 21 EMLINK errors,
and passes on local disk at the same seed. At seed {{3635CC89D23D0929}} it
passes on both, so the failure is seed-dependent.
These are one run per side per seed, so this is strong evidence for a host
cause rather than proof. No code change has been identified for these three
methods. A clean local-disk smoke run of RC3 has not completed yet, because the
host restarted during the runs.
h2. Related RC3 smoke failures
RecoveryAfterSoftCommitTest is SOLR-18530 (fix in PR #5098) and
TestCoordinatorRole.testNRTRestart is SOLR-16630 (fix in PR #5106); neither fix
is in RC3. testNRTRestart also fails on NFS with EMLINK and passes on local
disk at seed {{787C6229591C6F1E}}. This ticket covers the TlogReplica methods.
🤖 _AI text above_ 🤖 _(reviewed by Nick Shanin)_
> TestTlogReplica testKillLeader, testRecovery and testOnlyLeaderIndexes fail
> repeatedly in the 10.1.0 RC3 smoke tests
> --------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-18586
> URL: https://issues.apache.org/jira/browse/SOLR-18586
> Project: Solr
> Issue Type: Test
> Reporter: Nick Shanin
> Priority: Major
>
> h2. What was seen
> In five completed runs of the Solr 10.1.0 RC3 smoke tester (artifacts rev
> {{749712b71940c9f96d824c533f32f5c56dcd3924}}, JDK 21.0.12.1, aarch64), the
> build failed every time. {{TestTlogReplica.testKillLeader}} and
> {{TestTlogReplica.testRecovery}} failed in all five runs and
> {{testOnlyLeaderIndexes}} in four of the five, across different random seeds
> (for example {{787C6229591C6F1E}}, {{8D8640B367EDAD08}}, {{5AB00AEB1AE2392}}).
> h2. Host finding
> Matched pairs use the same seed and host; only the test temp directory
> differs (NFS {{/workspace}} vs local {{/tmp}}):
> * {{testKillLeader}} and {{testRecovery}}: fail on NFS with {{Too many
> links}} (EMLINK) from the hard links in {{IndexFetcher.java:1184}}
> ({{Files.createLink}}), and pass on local disk at the same seed.
> * {{testOnlyLeaderIndexes}} at seed {{5EC47295A32A8342}} (from run 005):
> fails on NFS with "Some replicas are not in sync with leader" and 21 EMLINK
> errors, and passes on local disk at the same seed. At seed
> {{3635CC89D23D0929}} it passes on both, so the failure is seed-dependent.
> These are one run per side per seed, so this is strong evidence for a host
> cause rather than proof. No code change has been identified for these three
> methods. A clean local-disk smoke run of RC3 has not completed yet, because
> the host restarted during the runs.
> h2. Related RC3 smoke failures
> RecoveryAfterSoftCommitTest is SOLR-18530 (fix in PR #5098) and
> TestCoordinatorRole.testNRTRestart is SOLR-16630 (fix in PR #5106); neither
> fix is in RC3. testNRTRestart also fails on NFS with EMLINK and passes on
> local disk at seed {{787C6229591C6F1E}}. This ticket covers the TlogReplica
> methods.
> h2. Fix branch (not a PR yet)
> https://github.com/nick-boss-tech/solr/tree/emlink-copy-fallback (head
> {{23053d08816}})
> * {{fdb785484c6}}: {{IndexFetcher}} falls back to {{Files.copy}} when
> {{Files.createLink}} fails (EMLINK on NFS).
> * {{23053d08816}}: JS client install uses {{--include=dev}} so the build does
> not depend on {{NODE_ENV}}. Unrelated to SOLR-18586.
> Tested on NFS at the seeds above: {{TestTlogReplica}},
> {{TestCoordinatorRole}} and the replication test group pass with zero EMLINK
> errors. The full RC3 smoke run on this head has not completed a clean gate
> yet (the final binary-release step has no result). No PR is open.
> 🤖 _AI text above_ 🤖 _(reviewed by Nick Shanin)_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
