I had blockdb in ssd, with 3 OSDs per host (8G ram) and the default 3G
bluestore_cache_size_ssd
I stopped having inconsistencies dropping the cache to 1G.
On 27/08/18 23:32, Tyler Bishop wrote:
Having a constant segfault issue under io load with my newly created
bluestore deployment.
https://pastebin.com/82YjXRm7
Setup is 28GB SSD LVM for block.db and 6T spinner for data.
Config:
[global]
fsid = REDACTED
mon_initial_members = cephmon-1001, cephmon-1002, cephmon-1003
mon_host = 10.20.142.5,10.20.142.6,10.20.142.7
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
# Fixes issue where image is created with newer than supported
features enabled.
rbd_default_features = 3
# Debug Tuning
debug_lockdep = 0/0
debug_context = 0/0
debug_crush = 0/0
debug_buffer = 0/0
debug_timer = 0/0
debug_filer = 0/0
debug_objecter = 0/0
debug_rados = 0/0
debug_rbd = 0/0
debug_journaler = 0/0
debug_objectcatcher = 0/0
debug_client = 0/0
debug_osd = 0/0
debug_optracker = 0/0
debug_objclass = 0/0
debug_filestore = 0/0
debug_journal = 0/0
debug_ms = 0/0
debug_monc = 0/0
debug_tp = 0/0
debug_auth = 0/0
debug_finisher = 0/0
debug_heartbeatmap = 0/0
debug_perfcounter = 0/0
debug_asok = 0/0
debug_throttle = 0/0
debug_mon = 0/0
debug_paxos = 0/0
debug_rgw = 0/0
[osd]
osd_mkfs_type = xfs
osd_mount_options_xfs =
rw,noatime,,nodiratime,inode64,logbsize=256k,delaylog
osd_mkfs_options_xfs = -f -i size=2048
osd_journal_size = 10240
filestore_queue_max_ops=1000
filestore_queue_max_bytes = 1048576000
filestore_max_sync_interval = 10
filestore_merge_threshold = 500
filestore_split_multiple = 100
osd_op_shard_threads = 6
journal_max_write_entries = 5000
journal_max_write_bytes = 1048576000
journal_queueu_max_ops = 3000
journal_queue_max_bytes = 1048576000
ms_dispatch_throttle_bytes = 1048576000
objecter_inflight_op_bytes = 1048576000
public network = 10.20.142.0/24 <http://10.20.142.0/24>
cluster_network = 10.20.136.0/24 <http://10.20.136.0/24>
osd_disk_thread_ioprio_priority = 7
osd_disk_thread_ioprio_class = idle
osd_max_backfills = 2
osd_recovery_sleep = 0.10
[client]
rbd_cache = False
rbd cache size = 33554432
rbd cache target dirty = 16777216
rbd cache max dirty = 25165824
rbd cache max dirty age = 2
rbd cache writethrough until flush = false
--------
2018-08-28 02:31:30.961954 7f64a895a700 4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/flush_job.cc:319]
[default] [JOB 19] Level-0 flush table #688: 6121532 bytes OK
2018-08-28 02:31:30.962476 7f64a895a700 4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_files.cc:242]
adding log 681 to recycle list
2018-08-28 02:31:30.962495 7f64a895a700 4 rocksdb: (Original Log Time
2018/08/28-02:31:30.961973)
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/memtable_list.cc:360]
[default] Level-0 commit table #688 started
2018-08-28 02:31:30.962501 7f64a895a700 4 rocksdb: (Original Log Time
2018/08/28-02:31:30.962413)
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/memtable_list.cc:383]
[default] Level-0 commit table #688: memtable #1 done
2018-08-28 02:31:30.962505 7f64a895a700 4 rocksdb: (Original Log Time
2018/08/28-02:31:30.962432) EVENT_LOG_v1 {"time_micros":
1535423490962423, "job": 19, "event": "flush_finished", "lsm_state":
[1, 4, 1, 0, 0, 0, 0], "immutable_memtables": 0}
2018-08-28 02:31:30.962509 7f64a895a700 4 rocksdb: (Original Log Time
2018/08/28-02:31:30.962458)
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_compaction_flush.cc:132]
[default] Level summary: base level 1 max bytes base 268435456 files[1
4 1 0 0 0 0] max score 0.84
2018-08-28 02:31:30.962517 7f64a895a700 4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_files.cc:388]
[JOB 19] Try to delete WAL files size 258068015, prev total WAL file
size 260608480, number of live WAL files 2.
2018-08-28 02:32:06.102335 7f64b917b700 4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_write.cc:684]
reusing log 681 from recycle list
2018-08-28 02:32:06.102473 7f64b917b700 4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_write.cc:725]
[default] New memtable created with log file: #689. Immutable
memtables: 0.
2018-08-28 02:32:06.102542 7f64a895a700 4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_compaction_flush.cc:49]
[JOB 20] Syncing log #687
2018-08-28 02:32:06.103394 7f64a895a700 4 rocksdb: (Original Log Time
2018/08/28-02:32:06.102527)
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_compaction_flush.cc:1158]
Calling FlushMemTableToOutputFile with column family [default], flush
slots available 1, compaction slots allowed 1, compaction slots
scheduled 1
2018-08-28 02:32:06.103407 7f64a895a700 4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/flush_job.cc:264]
[default] [JOB 20] Flushing memtable with next log file: 689
2018-08-28 02:32:06.103435 7f64a895a700 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1535423526103422, "job": 20, "event": "flush_started",
"num_memtables": 1, "num_entries": 97689, "num_deletes": 21335,
"memory_usage": 260069984}
2018-08-28 02:32:06.103446 7f64a895a700 4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/flush_job.cc:293]
[default] [JOB 20] Level-0 flush table #690: started
2018-08-28 02:32:06.155755 7f64a895a700 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1535423526155726, "cf_name": "default", "job": 20,
"event": "table_file_creation", "file_number": 690, "file_size":
6343137, "table_properties": {"data_size": 6153638, "index_size":
65232, "filter_size": 123278, "raw_key_size": 2289031,
"raw_average_key_size": 52, "raw_value_size": 5374531,
"raw_average_value_size": 122, "num_data_blocks": 1047, "num_entries":
43785, "filter_policy_name": "rocksdb.BuiltinBloomFilter",
"kDeletedKeys": "21429", "kMergeOperands": "220"}}
2018-08-28 02:32:06.155776 7f64a895a700 4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/flush_job.cc:319]
[default] [JOB 20] Level-0 flush table #690: 6343137 bytes OK
2018-08-28 02:32:06.156214 7f64a895a700 4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_files.cc:242]
adding log 687 to recycle list
2018-08-28 02:32:06.156225 7f64a895a700 4 rocksdb: (Original Log Time
2018/08/28-02:32:06.155790)
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/memtable_list.cc:360]
[default] Level-0 commit table #690 started
2018-08-28 02:32:06.156229 7f64a895a700 4 rocksdb: (Original Log Time
2018/08/28-02:32:06.156164)
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/memtable_list.cc:383]
[default] Level-0 commit table #690: memtable #1 done
2018-08-28 02:32:06.156239 7f64a895a700 4 rocksdb: (Original Log Time
2018/08/28-02:32:06.156178) EVENT_LOG_v1 {"time_micros":
1535423526156172, "job": 20, "event": "flush_finished", "lsm_state":
[2, 4, 1, 0, 0, 0, 0], "immutable_memtables": 0}
2018-08-28 02:32:06.156244 7f64a895a700 4 rocksdb: (Original Log Time
2018/08/28-02:32:06.156199)
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_compaction_flush.cc:132]
[default] Level summary: base level 1 max bytes base 268435456 files[2
4 1 0 0 0 0] max score 0.84
2018-08-28 02:32:06.156252 7f64a895a700 4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_files.cc:388]
[JOB 20] Try to delete WAL files size 257866117, prev total WAL file
size 259275521, number of live WAL files 2.
_____________________________________________
*Tyler Bishop*
EST 2007
O:513-299-7108 x1000
M:513-646-5809
http://BeyondHosting.net <http://beyondhosting.net/>
This email is intended only for the recipient(s) above and/or
otherwise authorized personnel. The information contained herein and
attached is confidential and the property of Beyond Hosting. Any
unauthorized copying, forwarding, printing, and/or disclosing
any information related to this email is prohibited. If you received
this message in error, please contact the sender and destroy all
copies of this email and any attachment(s).
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Alfredo Daniel Rezinovsky
Director de Tecnologías de Información y Comunicaciones
Facultad de Ingeniería - Universidad Nacional de Cuyo
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
