We have a kubernetes cluster that runs many small Solr installations in
separate deployments, each having a single pod containing standalone
Solr. Each one has a small volume attached for persistent storage.
Today I had a weird problem. There was only 1 worker node for a long
time as the number of deployments were few, but as we created more
deployments, the cluster scaled to 2 nodes. The Solr pods running on the
first node had no problems but the ones on the second node were
answering to http requests super slowly (/solr/admin/cores endpoint), I
mean fastest request took 30 seconds and some of them took even 120
seconds, this is all in the same network by the way (AWS vpc where
everything is in the same availability zone) and index is empty (no
cores or data).
After doing many diagnostics I tried my luck disabling the security
manager by setting the env variable:
- name: SOLR_SECURITY_MANAGER_ENABLED
value: "false"
in the deployment descriptor which solved the problem, now all pods
answer fast whether they are on the second k8s node or on the first
one...
My question is how can this be and is it ok to just disable it, I mean
since it's related to security, can I set it in a better way without
completely disabling it?
-ufuk yilmaz
