...
--
With best regards,
Vitaliy Filippov
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
It seems no, I've just tested it on another small cluster with HDDs only -
no change
Does it make sense to test disabling this on hdd cluster only?
--
With best regards,
Vitaliy Filippov
___
ceph-users mailing list
ceph-users@lists.ceph.com
http
cache (although both are SSD-related
as I understand).
Maybe your HDD are connected via some RAID controller and when you disable
cache it doesn't really get disabled, but the kernels just stops to issue
flush requests and makes some writes unsafe?
--
With best regards,
Vitaliy Filippov
Try lspci -vs and look for
`Capabilities: [148] Device Serial Number 00-02-c9-03-00-4f-68-7e`
in the output
--
With best regards,
Vitaliy Filippov
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users
ndom write iops by an order of magnitude. not sure
why. maybe because kernel flushes disk queue on every sync if it thinks
disk cache is enabled...
--
With best regards,
Vitaliy Filippov
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
uate disk handles fsync's (i.e SATA
FLUSH CACHE commands). So transactional writes should never be lost, and
in Ceph ALL writes are transactional - Ceph issues fsync's all the time.
Another example is DBMS-es - they also issue an fsync when you COMMIT.
--
With be
for 4k random writes)
--
With best regards,
Vitaliy Filippov
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
,
Vitaliy Filippov
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
a following error message:
# rbd map image01 --pool testbench --name client.admin
You don't need to map it to run benchmarks, use `fio --ioengine=rbd`
(however you'll still need /etc/ceph/ceph.client.admin.keyring)
--
With best regards,
Vitaliy Filippov
...which only works when mapped with `virtio-scsi` (not with the regular
virtio driver) :)
The only important thing is to enable discard/trim on the file system.
--
With best regards,
Vitaliy Filippov
___
ceph-users mailing list
ceph-users
ly I moved
them to SSDs with rebased Igor Fedotov's ceph-bluestool ... oops :)
ceph-bluestore-tool. Although I still don't understand where the number 3
comes from? Ceph's default bluestore_rocksdb_options states there are
4*256MB memtables, it's 1GB, not 3...
--
With best re
ything in-between those
sizes are pointless. Only ~3GB of SSD will ever be used out of a
28GB partition. Likewise a 240GB partition is also pointless as only
~30GB will be used.
Where did you get those numbers? I would like to read more if you can
point to a link.
--
With best regards,
Vitaliy Fi
interaction isn't the
bottleneck currently, it's Ceph code itself. I also tried once, but I
couldn't make it work. When I have some spare NVMe's I'll make another
attempt.
So... try it and share your results here :) we're all interested.
--
With best regards,
Vitaliy Filippov
of nvmes with Ceph is to
create multiple partitions on the nvme with an osd on each partition.
That
way you get more osd processes and CPU per nvme device. I've heard of
people using up to 4 partitions like this.
--
With best regards,
Vitaliy Filippov
I think this should not lead to blocked ops in any case, even if the
performance is low...
--
With best regards,
Vitaliy Filippov
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
regards,
Vitaliy Filippov
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
!
--
With best regards,
Vitaliy Filippov
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
eliminate such behaviour?
--
With best regards,
Vitaliy Filippov
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
/test --size=10G --runtime=60 --group_reporting
--name=test --direct=1
I get 36000 iop/s on bluestore while having 11500 on filestore.
Using randwrite gives me 17000 on filestore and only 9500 on bluestore.
This is on all flash / ssd running luminous 12.2.10.
--
With best regards,
Vitaliy
owhere near native, asking if this was normal. But never got
a response to it. I can remember that they send everyone a questionaire
and asked if they should focus on performance more, now I wished I
checked that box ;)
--
With best regards,
Vitaliy Filippov
___
ce
+1, I also think's it's strange that deleting OSD by "osd out -> osd
purge" causes two rebalances instead of one.
--
With best regards,
Vitaliy Filippov
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/list
mber is around 15000.
Create multiple OSDs on a single NVMe and sacrifice your CPU usage if you
want better results.
--
With best regards,
Vitaliy Filippov
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
elf.
--
With best regards,
Vitaliy Filippov
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
with the
number of OSDs at all, but it's usually what real applications like DBMSes
need.
The second measures your max possible random write throughput which you
probably won't be able to utilize if you don't have enough VMs all writing
in parallel.
--
With best regards,
Vitaliy Filippov
hat if
one day fio gets it enabled? :)
how about adding: --sync=1 --numjobs=1 to the command as well?
--
With best regards,
Vitaliy Filippov
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
would never see in production in the
tests does not really help my understanding. I think a concurrency level
of 16 is in the top of what I would expect our PostgreSQL databases to do
in real life.
--
With best regards,
Vitaliy Filippov
___
ceph-users
verall
transport performance.
--
Roman
--
With best regards,
Vitaliy Filippov
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
f 4K random
writes.
--
With best regards,
Vitaliy Filippov
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
oints")? :)
--
With best regards,
Vitaliy Filippov
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
In our russian-speaking Ceph chat we swear "ceph inside kuber" people all
the time because they often do not understand in what state their cluster
is at all
// Sorry to intervene :))
--
With best regards,
Vitaliy Filippov
___
ceph-use
All values except 4, 30 and 286 GB are currently useless in ceph with
default rocksdb settings :)
That's what you are seeing - all devices just use ~28 GB and everything
else goes to HDDs.
--
With best regards,
Vitaliy Filippov
___
ceph-users
22
44 4444
--
With best regards,
Vitaliy Filippov
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
One RAID0 array per drive :)
I can't understand how using RAID0 is better than JBOD, considering jbod
would be many individual disks, each used as OSDs, instead of a single
big one used as a single OSD.
--
With best regards,
Vitaliy Filippov
...
--
With best regards,
Vitaliy Filippov
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
you actually get better random writes
on HDDs with bluestore, too.
--
With best regards,
Vitaliy Filippov
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
=rpool
-runtime=60 -rbdname=testimg
--
With best regards,
Vitaliy Filippov
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
It helps performance, but it can also lead to data loss if the raid
controller is crap (not flushing data correctly)
--
With best regards,
Vitaliy Filippov
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi
OK, I meant "it may help performance" :) the main point is that we had at
least one case of data loss due to some Adaptec controller in RAID0 mode
discussed recently in our ceph chat...
--
With best regards,
Vitaliy Filippov
___
SD would
suffice?
We were thinking of using SATA Read Intensive 6Gbps 1DWPD SSDs.
Does someone has some experience with this configuration? Would we need
SAS ssds instead of SATA? And Mixed Use 3WPD instead of Read intensive?
--
With best regards,
Vitaliy Filippov
.
--
With best regards,
Vitaliy Filippov
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
rados bench -p scbench 60 seq --io-size 8192 --io-threads 256
Read size:4194304
rados bench doesn't have --io-size option
testing sequential read with 8K I/O size is a strange idea anyway though
--
With best regards,
Vitaliy Filippov
41 matches
Mail list logo
