Solr Collection reload task has been in running state for a very long time

[Moulay Hicham]

Hi,

We have a solr cluster of 30 nodes with a Replication Factor =3.
Each index size is about 80GB.
Solr version is 8.1
The cluster has high TPS both in read and write.

We have recently made a schema change and uploaded it using ZKCLI
script. Then we issue a collection reload async request:
admin/collections?action=RELOAD&name=<collectionname>&async=1000'

When we check on the status of this request, it shows that it's still running:

admin/collections?action=REQUESTSTATUS&requestid=1000'
{
  "responseHeader":{
    "status":0,
    "QTime":1},
  "status":{
    "state":"running",
    "msg":"found [1000] in running tasks"}}

This task has been in a running state for about 5 hours so far. I am
not sure if this is expected or the status of this task failed or
completed but never reported back to zookeeper.

Also if running for that long - is it because the index is being
actively (with high TPS) updated? We have a softcommit of 10s and
hadcommit of 60s.

Please help me understand what's going on.

Thanks,
Moulay