Hello,
I have been running Solr (5.5.0) on HDFS.
Recently a collection just went missing with all the instanceDirs and
Datadirs getting deleted. The following logs in the solrCloud overseer.
2017-01-20 20:42:39,515 INFO org.apache.solr.core.SolrCore:
[3044_01_17_shard4_replica1] CLOSING SolrCore
org.apache.solr.core.SolrCore@2e2e0a23
2017-01-20 20:42:39,665 INFO org.apache.solr.core.SolrCore:
[3044_01_17_shard4_replica1] Closing main searcher on request.
2017-01-20 20:42:39,690 INFO org.apache.solr.core.CachingDirectoryFactory:
looking to close hdfs://Ingest/solr53/3044_01_17/core_node3/data
[CachedDir<<refCount=0;path=hdfs://Ingest/solr53/3044_01_17/core_node3/data;done=false>>]
2017-01-20 20:42:39,690 INFO org.apache.solr.core.CachingDirectoryFactory:
looking to close hdfs://Ingest/solr53/3044_01_17/core_node3/data/index
[CachedDir<<refCount=0;path=hdfs://Ingest/solr53/3044_01_17/core_node3/data/index;done=false>>,
CachedDir<<refCount=0;path=hdfs://Ingest/solr53/3044_01_17/core_node3/data;done=false>>]
2017-01-20 20:42:39,690 INFO org.apache.solr.core.CachingDirectoryFactory:
Closing directory: hdfs://Ingest/solr53/3044_01_17/core_node3/data/index
2017-01-20 20:42:39,712 INFO org.apache.solr.store.hdfs.HdfsDirectory:
Closing hdfs directory hdfs://Ingest/solr53/3044_01_17/core_node3/data/index
2017-01-20 20:42:39,713 INFO org.apache.solr.core.CachingDirectoryFactory:
Closing directory: hdfs://Ingest/solr53/3044_01_17/core_node3/data
2017-01-20 20:42:39,713 INFO org.apache.solr.store.hdfs.HdfsDirectory:
Closing hdfs directory hdfs://Ingest/solr53/3044_01_17/core_node3/data
2017-01-20 20:42:39,713 INFO org.apache.solr.core.CachingDirectoryFactory:
looking to close hdfs://Ingest/solr53/3044_01_17/core_node3/data []
2017-01-20 20:42:39,713 INFO org.apache.solr.core.CachingDirectoryFactory:
Removing directory after core close:
hdfs://Ingest/solr53/3044_01_17/core_node3/data
"core":"3044_01_17_shard13_replica1",
"collection":"3044_01_17",
2017-01-20 20:42:39,780 INFO org.apache.solr.cloud.overseer.ZkStateWriter:
going to update_collection /collections/3044_01_17/state.json version: 2164
2017-01-20 20:42:39,782 INFO org.apache.solr.common.cloud.ZkStateReader: A
cluster state change: [WatchedEvent state:SyncConnected
type:NodeDataChanged path:/collections/3044_01_17/state.json] for
collection [3044_01_17] has occurred - updating... (live nodes size: [85])
2017-01-20 20:42:39,783 INFO org.apache.solr.common.cloud.ZkStateReader:
Updating data for [3044_01_17] from [2164] to [2165]
"core":"3044_01_17_shard32_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard22_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard27_replica1",
"collection":"3044_01_17",
2017-01-20 20:42:39,822 INFO
org.apache.solr.cloud.ShardLeaderElectionContextBase: Removing leader
registration node on cancel: /collections/3044_01_17/leaders/shard4/leader 9
2017-01-20 20:42:39,832 INFO org.apache.solr.cloud.ElectionContext:
Canceling election
/collections/3044_01_17/leader_elect/shard4/election/241183598302995297-core_node3-n_0000000008
2017-01-20 20:42:39,833 INFO org.apache.solr.common.cloud.ZkStateReader:
Removing watch for uninteresting collection [3044_01_17]
"core":"3044_01_17_shard4_replica1",
"collection":"3044_01_17",
2017-01-20 20:42:39,852 INFO org.apache.solr.servlet.HttpSolrCall: [admin]
webapp=null path=/admin/cores
params={deleteInstanceDir=true&action=UNLOAD&core=3044_01_17_shard4_replica1&wt=javabin&qt=/admin/cores&deleteDataDir=true&version=2}
status=0 QTime=341
2017-01-20 20:42:39,938 INFO org.apache.solr.cloud.overseer.ZkStateWriter:
going to update_collection /collections/3044_01_17/state.json version: 2165
2017-01-20 20:42:39,940 INFO org.apache.solr.common.cloud.ZkStateReader:
Uninteresting collection [3044_01_17]
"core":"3044_01_17_shard14_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard35_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard11_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard3_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard1_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard2_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard5_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard42_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard28_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard45_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard8_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard49_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard9_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard34_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard25_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard7_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard10_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard36_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard47_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard6_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard43_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard26_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard40_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard41_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard33_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard38_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard39_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard18_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard15_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard31_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard23_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard16_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard17_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard30_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard20_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard50_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard48_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard12_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard24_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard46_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard44_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard37_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard29_replica1",
"collection":"3044_01_17",
"core":"3044_01_17_shard21_replica1",
"collection":"3044_01_17",
2017-01-20 20:42:40,524 INFO org.apache.solr.cloud.overseer.ZkStateWriter:
going to update_collection /collections/3044_01_17/state.json version: 2166
"core":"3044_01_17_shard19_replica1",
"collection":"3044_01_17",
"name":"3044_01_17"} current state version: 0
2017-01-20 20:42:41,422 INFO org.apache.solr.cloud.overseer.ZkStateWriter:
going to delete_collection /collections/3044_01_17/state.json
Please provide some insights.
