Re: [ceph-users] 403-Forbidden error using radosgw
Hi Lakshmi, Is your issues solved, can you please let me know if you solved this, bcoz I am also having same issue. Thanks Regards, Naga Venkata ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] v0.80.10 Firefly released
This is a bugfix release for Firefly.
We recommend that all Firefly users upgrade at their convenience.
Notable Changes
---
* rgw: check for timestamp for s3 keystone auth (#10062, Abhishek Lekshmanan)
* mon: PGMonitor: several stats output error fixes (#10257, Joao Eduardo Luis)
* osd: pg stuck stale after create with activation delay (#11197, Samuel Just)
* mon: OSDMonitor: fallback to json-pretty in case of invalid formatter (#9538,
Loic Dachary)
* osd: ENOENT on clone (#11199, Samuel Just)
* doc: Fix ceph command manpage to match ceph -h (#10676, David Zafman)
* rgw: Swift API. Support for X-Remove-Container-Meta-{key} header. (#10475,
Dmytro Iurchenko)
* rgw: fail s3 POST auth if keystone not configured (#10698, Yehuda Sadeh)
* mds: fix assertion caused by system clock backwards (#11053, Yan, Zheng)
* osd: check that source OSD is valid for MOSDRepScrub (#9555, Sage Weil)
* osd: requeue blocked op before flush it was blocked on (#10512, Sage Weil)
* osd: ReplicatedPG: fail a non-blocking flush if the object is being scrubbed
(#8011, Samuel Just)
* tools: Fix do_autogen.sh so that -L is allowed (#11303, Alfredo Deza)
* tools: ceph-objectstore-tool: Use exit status 11 for incompatible import
attemp... (#11139, David Zafman)
* fs: mount.ceph: avoid spurious error message (#10351, Yan, Zheng)
* tests: force checkout of submodules (#11157, Loic Dachary)
* osd: FAILED assert(soid scrubber.start || soid = scrubber.end) (#11156,
Samuel Just)
* osd: cache tiering: fix the atime logic of the eviction (#9915, Zhiqiang Wang)
* tools: ceph-objectstore-tool: Output only unsupported features when
incomatible (#11176, David Zafman)
* osd: PG::all_unfound_are_queried_or_lost for non-existent osds (#10976,
Mykola Golub)
* rgw: Swift API. Dump container's custom metadata. (#10665, Dmytro Iurchenko)
* osd: cancel_pull: requeue waiters (#11244, Samuel Just)
* librbd: snap_remove should ignore -ENOENT errors (#3, Jason Dillaman)
* common: SimpleMessenger: allow RESETSESSION whenever we forget an endpoint
(#10080, Greg Farnum)
* osd: load_pgs: we need to handle the case where an upgrade from earlier
versions which ignored non-existent pgs resurrects a pg with a prehistoric
osdmap (#11429, Samuel Just)
* rgw: merge manifests correctly when there's prefix override (#11622, Yehuda
Sadeh)
* mon: ignore osd failures from before up_from (#10762, Sage Weil)
* common: WorkQueue: make wait timeout on empty queue configurable (#10817,
Samuel Just)
* osd: DBObjectMap: lock header_lock on sync() (#9891, Samuel Just)
* rgw: civetweb should use unique request id (#11720, Orit Wasserman)
* librados: fix resources leakage in RadosClient::connect() (#10425, Radoslaw
Zarzynski)
* osd: fix negative degraded objects during backfilling (#7737, Guang Yang)
* osd: FileJournal: fix journalq population in do_read_entry() (#6003, Samuel
Just)
* common: Rados.shutdown() dies with Illegal instruction (core dumped) (#10153,
Federico Simoncelli)
* osd: ReplicatedPG::scan_range: an object can disappear between the list and
t... (#10150, Samuel Just)
* osd: PGLog: include rollback_info_trimmed_to in (read|write)_log (#10157,
Samuel Just)
* osd: PG: always clear_primary_state (#10059, Samuel Just)
* crush: set_choose_tries = 100 for erasure code rulesets (#10353, Loic Dachary)
* build/ops: ceph.spec.in: package mkcephfs on EL6 (#11955, Ken Dreyer)
* doc: backport ceph-disk man page to Firefly (#10724, Nilamdyuti Goswami)
* librados: Fix memory leak in python rados bindings (#10723, Josh Durgin)
* osd: ReplicatedPG::on_change: clean up callbacks_for_degraded_object (#8753,
Samuel Just)
* librados: Translate operation flags from C APIs (#10497, Matt Richards)
* common: messages/MWatchNotify: include an error code in the message (#9193,
Sage Weil)
* mon: Monitor: fix timecheck rounds period (#10546, Joao Eduardo Luis)
* mon: OSDMonitor: do not trust small values in osd epoch cache (#10787, Sage
Weil)
* rgw: keystone token cache does not work correctly (#11125, Yehuda Sadeh)
* librbd: acquire cache_lock before refreshing parent (#5488, Jason Dillaman)
* objecter: fix map skipping (#9986, Ding Dinghua)
* osd: get the currently atime of the object in cache pool for eviction (#9985,
Sage Weil)
* rgw: don't allow negative / invalid content length (#11890, Yehuda Sadeh)
* osd: ObjectStore: Don't use largest_data_off to calc data_align. (#10014,
Jianpeng Ma)
* osd: osd_types: op_queue_age_hist and fs_perf_stat should be in
osd_stat_t::o... (#10259, Samuel Just)
* osd: erasure-code-profile set races with erasure-code-profile rm (#11144,
Loic Dachary)
* build/ops: debian: ceph-test and rest-bench debug packages should require
their respective binary packages (#11673, Ken Dreyer)
* mon: MonCap: take EntityName instead when expanding profiles (#10844, Joao
Eduardo Luis)
* rgw: use correct objv_tracker for bucket instance (#11416, Yehuda Sadeh)
* rgw: generate new tag for object when setting object attrs (#11256, Yehuda
[ceph-users] Ceph Tech Talk next week
Hey cephers, Just a reminder that the Ceph Tech Talk on CephFS that was scheduled for last month (and cancelled due to technical difficulties) has been rescheduled for this month's talk. It will be happening next Thurs at 17:00 UTC (1p EST) on our Blue Jeans conferencing system. If you have any questions feel free to let me know. Thanks. http://ceph.com/ceph-tech-talks/ -- Best Regards, Patrick McGarry Director Ceph Community || Red Hat http://ceph.com || http://community.redhat.com @scuttlemonkey || @ceph ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Performance dégradation after upgrade to hammer
Hi Mark I've something like 600 write IOPs on EC pool and 800 write IOPs on replicated 3 pool with rados bench With Radosgw I have 30/40 write IOPs with Cosbench (1 radosgw- the same with 2) and servers are sleeping : - 0.005 core for radosgw process - 0.01 core for osd process I don't know if we can have .rgw* pool locking or something like that with Hammer (or situation specific to me) On 100% read profile, Radosgw and Ceph servers are working very well with more than 6000 IOPs on one radosgw server : - 7 cores for radosgw process - 1 core for each osd process - 0,5 core for each Apache process Thanks Sent from my iPhone On 14 juil. 2015, at 21:03, Mark Nelson mnel...@redhat.com wrote: Hi Florent, 10x degradation is definitely unusual! A couple of things to look at: Are 8K rados bench writes to the rgw.buckets pool slow? You can with something like: rados -p rgw.buckets bench 30 write -t 256 -b 8192 You may also want to try targeting a specific RGW server to make sure the RR-DNS setup isn't interfering (at least while debugging). It may also be worth creating a new replicated pool and try writes to that pool as well to see if you see much difference. Mark On 07/14/2015 07:17 PM, Florent MONTHEL wrote: Yes of course thanks Mark Infrastructure : 5 servers with 10 sata disks (50 osd at all) - 10gb connected - EC 2+1 on rgw.buckets pool - 2 radosgw RR-DNS like installed on 2 cluster servers No SSD drives used We're using Cosbench to send : - 8k object size : 100% read with 256 workers : better results with Hammer - 8k object size : 80% read - 20% write with 256 workers : real degradation between Firefly and Hammer (divided by something like 10) - 8k object size : 100% write with 256 workers : real degradation between Firefly and Hammer (divided by something like 10) Thanks Sent from my iPhone On 14 juil. 2015, at 19:57, Mark Nelson mnel...@redhat.com wrote: On 07/14/2015 06:42 PM, Florent MONTHEL wrote: Hi All, I've just upgraded Ceph cluster from Firefly 0.80.8 (Redhat Ceph 1.2.3) to Hammer (Redhat Ceph 1.3) - Usage : radosgw with Apache 2.4.19 on MPM prefork mode I'm experiencing huge write performance degradation just after upgrade (Cosbench). Do you already run performance tests between Hammer and Firefly ? No problem with read performance that was amazing Hi Florent, Can you talk a little bit about how your write tests are setup? How many concurrent IOs and what size? Also, do you see similar problems with rados bench? We have done some testing and haven't seen significant performance degradation except when switching to civetweb which appears to perform deletes more slowly than what we saw with apache+fcgi. Mark Sent from my iPhone ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Failed to deploy Ceph Hammer(0.94.2) MDS
Hi cephers,
I am using CephFS on AWS as persistent shared storage.
Last night, I upgraded to Hammer(v0.94.2) from Firefly, I cannot enable MDS
service.
Here is the log of MDS service:
2015-07-22 02:25:08.284564 7f8417d2c7c0 0 ceph version 0.94.2
(5fb85614ca8f354284c713a2f9c610860720bbf3), process ceph-mds, pid 817
starting mds.node-01 at :/0
2015-07-22 02:25:08.288355 7f8417d2c7c0 -1 mds.-1.0 log_to_monitors
{default=true}
I cannot figure out anything from this log, such that I trun to monitor's
log.
Here is the log of Monitor:
...
2015-07-22 02:25:07.163649 7f1f0765d700 0 mon.node-01@0(leader) e1
handle_command mon_command({prefix: auth get-or-create, entity:
mds.vos-node-01, caps: [mds, allow, osd, allow *, mon, allow
*]} v 0) v1
2015-07-22 02:25:07.163686 7f1f0765d700 0 log_channel(audit) log [INF] :
from='client.4142 :/0' entity='client.admin' cmd=[{prefix: auth
get-or-create, entity: mds.node-01, caps: [mds, allow, osd,
allow *, mon, allow *]}]: dispatch
2015-07-22 02:25:07.200885 7f1f08e60700 0 log_channel(audit) log [INF] :
from='client.4142 :/0' entity='client.admin' cmd='[{prefix: auth
get-or-create, entity: mds.node-01, caps: [mds, allow, osd,
allow *, mon, allow *]}]': finished
2015-07-22 02:25:07.379095 7f1f0765d700 1 mon.node-01@0(leader).mds e1
warning, MDS mds.? 10.0.10.111:6804/823 up but filesystem disabled
Any idea what is happening to MDS? and how to fix this?
Thanks,
Houwa Cheung
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph 0.94 (and lower) performance on 1 hosts ??
Hi Frederic, When you have Ceph cluster with 1 node you don’t experienced network and communication overhead due to distributed model With 2 nodes and EC 4+1 you will have communication between 2 nodes but you will keep internal communication (2 chunks on first node and 3 chunks on second node) On your configuration EC pool is setup with 4+1 so you will have for each write overhead due to write spreading on 5 nodes (for 1 customer IO, you will experience 5 Ceph IO due to EC 4+1) It’s the reason for that I think you’re reaching performance stability with 5 nodes and more in your cluster On Jul 20, 2015, at 10:35 AM, SCHAER Frederic frederic.sch...@cea.fr wrote: Hi, As I explained in various previous threads, I’m having a hard time getting the most out of my test ceph cluster. I’m benching things with rados bench. All Ceph hosts are on the same 10GB switch. Basically, I know I can get about 1GB/s of disk write performance per host, when I bench things with dd (hundreds of dd threads) +iperf 10gbit inbound+iperf 10gbit outbound. I also can get 2GB/s or even more if I don’t bench the network at the same time, so yes, there is a bottleneck between disks and network, but I can’t identify which one, and it’s not relevant for what follows anyway (Dell R510 + MD1200 + PERC H700 + PERC H800 here, if anyone has hints about this strange bottleneck though…) My hosts each are connected though a single 10Gbits/s link for now. My problem is the following. Please note I see the same kind of poor performance with replicated pools... When testing EC pools, I ended putting a 4+1 pool on a single node in order to track down the ceph bottleneck. On that node, I can get approximately 420MB/s write performance using rados bench, but that’s fair enough since the dstat output shows that real data throughput on disks is about 800+MB/s (that’s the ceph journal effect, I presume). I tested Ceph on my other standalone nodes : I can also get around 420MB/s, since they’re identical. I’m testing things with 5 10Gbits/s clients, each running rados bench. But what I really don’t get is the following : - With 1 host : throughput is 420MB/s - With 2 hosts : I get 640MB/s. That’s surely not 2x420MB/s. - With 5 hosts : I get around 1375MB/s . That’s far from the expected 2GB/s. The network never is maxed out, nor are the disks or CPUs. The hosts throughput I see with rados bench seems to match the dstat throughput. That’s as if each additional host was only capable of adding 220MB/s of throughput. Compare this to the 1GB/s they are capable of (420MB/s with journals)… I’m therefore wondering what could possibly be so wrong with my setup ?? Why would it impact so much the performance to add hosts ? On the hardware side, I have Broadcam BCM57711 10-Gigabit PCIe cards. I know, not perfect, but not THAT bad neither… ? Any hint would be greatly appreciated ! Thanks Frederic Schaer ___ ceph-users mailing list ceph-users@lists.ceph.com mailto:ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph with SSD and HDD mixed
Hello Mario, in the end your workload defines which option(s) can be considered. They are different trade offs between read/write performance and the price That depend on your workload. E.g. - distribution of reads/writes - size of IO Requests (4k IO-Operations or 4MB..) - „locality“ of the IO-Operations (is there a small set of Data that is heavily used and other data more or less unused or. all stored data is used more or less equally). - required Bandwidth and latency ... Usually SSD for the OSD-Journal in a replicated pool with size=3 is a setup that works reasonably well for most applications. But for an optimized setup you’ll have to analyze your requirements and then fit the setup to your needs (or hire someone who helps you). greetings Johannes Am 22.07.2015 um 02:58 schrieb Mario Codeniera mario.codeni...@gmail.com: Hi Johannes, Thanks for your reply. I am naive for this, no idea how to make a configurations or where I can starts? based on the 4 options mentioned. Hope you can expound it further if possible. Best regards, Mario On Tue, Jul 21, 2015 at 2:44 PM, Johannes Formann mlm...@formann.de wrote: Hi, Can someone give an insights, if it possible to mixed SSD with HDD? on the OSD. you’ll have more or less four options: - SSDs for the journals of the OSD-process (SSD must be able to perform good on synchronous writes) - an SSD only pool for „high performance“ data - Using SSDs for the primary copy (fast reads), can be combined with the first - Using a cache pool with an SSD-only pool in front of the main disk-pool How can we speed up the uploading for file for example, as per experience it took around 18mins to load 20Gb images (via glance), in 1Gb network. Or it is just normal? That’s about 20MB/s, for (I guess) sequential writes on a disk only cluster that’s ok. But you can improve that with SSDs, but you have to choose the best option for your setup, depending on the expected workload. greetings Johannes ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] CephFS New EC Data Pool
(as per IRC) Yep, that's a bug alright. http://tracker.ceph.com/issues/12426
I expect we'll backport to hammer once fixed.
John
On 21/07/15 22:39, Adam Tygart wrote:
Hello all,
I'm trying to add a new data pool to CephFS, as we need some longer
term archival storage.
ceph mds add_data_pool archive
Error EINVAL: can't use pool 'archive' as it's an erasure-code pool
Here are the steps taken to create the pools for this new datapool:
ceph osd pool create arccache 512 512 replicated replicated_ruleset
ceph osd pool set arccache min_size 2
ceph osd pool set arccache size 3
ceph osd erasure-code-profile set ec62profile k=6 m=2
ruleset-failure-domain=disktype ruleset-root=std
ceph osd pool create archive 2048 2048 erasure ec62profile ecpool
ceph osd tier add-cache archive arccache $((1024*1024*1024*1024*5))
ceph osd tier cache-mode arccache writeback
ceph osd tier set-overlay archive arccache
ceph osd pool set arccache cache_target_dirty_ratio 0.3
ceph osd pool set arccache target_max_objects 200
I'm running Ceph 0.94.2 on CentOS 7.1
The other thing that is probably *not* what we want is that I can add
the cache tier (arccache) as a datapool to CephFS. Doing so adds pool
id 35 (the cache tier) to the mdsmap, which is not what happens when
you create a new cephfs with a tiered EC pool as the datapool.
dumped mdsmap epoch 63386
epoch 63386
flags 0
created 2015-06-19 09:52:52.598619
modified2015-07-21 16:21:12.672241
tableserver 0
root0
session_timeout 60
session_autoclose 300
max_file_size 17592186044416
last_failure63309
last_failure_osd_epoch 86152
compat compat={},rocompat={},incompat={1=base v0.20,2=client
writeable ranges,3=default file layouts on dirs,4=dir inode in
separate object,5=mds uses versioned encoding,6=dirfrag is stored in
omap,8=no anchor
table}
max_mds 1
in 0
up {0=142503496}
failed
stopped
data_pools 34,35
metadata_pool 32
inline_data disabled
141642223: 10.5.38.2:6800/78600 'hobbit02' mds.-1.0 up:standby seq 1
141732776: 10.5.38.14:6846/5875 'hobbit14' mds.-1.0 up:standby seq 1
156005649: 10.5.38.13:6892/20895 'hobbit13' mds.-1.0 up:standby seq 1
142503496: 10.5.38.1:6926/213073 'hobbit01' mds.0.2916 up:active seq 41344
Any thoughts? Is it a bug? Any work arounds?
--
Adam
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] client io doing unrequested reads
On 21/07/15 21:54, Hadi Montakhabi wrote:
Hello Cephers,
I am using CephFS, and running some benchmarks using fio.
After increasing the object_size to 33554432, when I try to run some
read and write tests with different block sizes, when I get to block
size of 64m and beyond, Ceph does not finish the operation (I tried
letting it run for more than a day at least three times).
However, when I cancel the job and I expect to see no io operations,
here is what I get:
Is the layout default, apart from the change to object_size?
What version are the client and server?
[cephuser@node01 ~]$ ceph -s
cluster b7beebf6-ea9f-4560-a916-a58e106c6e8e
health HEALTH_OK
monmap e3: 3 mons at
{node02=192.168.17.212:6789/0,node03=192.168.17.213:6789/0,node04=192.168.17.214:6789/0
http://192.168.17.212:6789/0,node03=192.168.17.213:6789/0,node04=192.168.17.214:6789/0}
election epoch 8, quorum 0,1,2 node02,node03,node04
mdsmap e74: 1/1/1 up {0=node02=up:active}
osdmap e324: 14 osds: 14 up, 14 in
pgmap v155699: 768 pgs, 3 pools, 15285 MB data, 1772 objects
91283 MB used, 7700 GB / 7817 GB avail
768 active+clean
* client io 2911 MB/s rd, 90 op/s*
If I do ceph -w, it shows me that it is constantly doing reads, but I
have no idea from where and when it would stop?
I had to remove my CephFS file system and the associated pools and
start things from scratch.
1. Any idea what is happening?
Not really... are you using the fuse client? Enabling debug objecter =
10 on the client will give you a log that says what writes the client
is doing.
2. When this happens, do you know a better way to get out of the
situation without destroying the filesystem and the pools?
This is probably a client issue, so I would expect killing the client to
get you out of it.
Cheers,
John
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph Tech Talk next week
On Tue, Jul 21, 2015 at 6:09 PM, Patrick McGarry pmcga...@redhat.com wrote: Hey cephers, Just a reminder that the Ceph Tech Talk on CephFS that was scheduled for last month (and cancelled due to technical difficulties) has been rescheduled for this month's talk. It will be happening next Thurs at 17:00 UTC (1p EST) So that's July 30, according to the website, right? :) ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] CephFS New EC Data Pool
Hello all,
I'm trying to add a new data pool to CephFS, as we need some longer
term archival storage.
ceph mds add_data_pool archive
Error EINVAL: can't use pool 'archive' as it's an erasure-code pool
Here are the steps taken to create the pools for this new datapool:
ceph osd pool create arccache 512 512 replicated replicated_ruleset
ceph osd pool set arccache min_size 2
ceph osd pool set arccache size 3
ceph osd erasure-code-profile set ec62profile k=6 m=2
ruleset-failure-domain=disktype ruleset-root=std
ceph osd pool create archive 2048 2048 erasure ec62profile ecpool
ceph osd tier add-cache archive arccache $((1024*1024*1024*1024*5))
ceph osd tier cache-mode arccache writeback
ceph osd tier set-overlay archive arccache
ceph osd pool set arccache cache_target_dirty_ratio 0.3
ceph osd pool set arccache target_max_objects 200
I'm running Ceph 0.94.2 on CentOS 7.1
The other thing that is probably *not* what we want is that I can add
the cache tier (arccache) as a datapool to CephFS. Doing so adds pool
id 35 (the cache tier) to the mdsmap, which is not what happens when
you create a new cephfs with a tiered EC pool as the datapool.
dumped mdsmap epoch 63386
epoch 63386
flags 0
created 2015-06-19 09:52:52.598619
modified2015-07-21 16:21:12.672241
tableserver 0
root0
session_timeout 60
session_autoclose 300
max_file_size 17592186044416
last_failure63309
last_failure_osd_epoch 86152
compat compat={},rocompat={},incompat={1=base v0.20,2=client
writeable ranges,3=default file layouts on dirs,4=dir inode in
separate object,5=mds uses versioned encoding,6=dirfrag is stored in
omap,8=no anchor
table}
max_mds 1
in 0
up {0=142503496}
failed
stopped
data_pools 34,35
metadata_pool 32
inline_data disabled
141642223: 10.5.38.2:6800/78600 'hobbit02' mds.-1.0 up:standby seq 1
141732776: 10.5.38.14:6846/5875 'hobbit14' mds.-1.0 up:standby seq 1
156005649: 10.5.38.13:6892/20895 'hobbit13' mds.-1.0 up:standby seq 1
142503496: 10.5.38.1:6926/213073 'hobbit01' mds.0.2916 up:active seq 41344
Any thoughts? Is it a bug? Any work arounds?
--
Adam
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] client io doing unrequested reads
Hello Cephers,
I am using CephFS, and running some benchmarks using fio.
After increasing the object_size to 33554432, when I try to run some read
and write tests with different block sizes, when I get to block size of 64m
and beyond, Ceph does not finish the operation (I tried letting it run for
more than a day at least three times).
However, when I cancel the job and I expect to see no io operations, here
is what I get:
[cephuser@node01 ~]$ ceph -s
cluster b7beebf6-ea9f-4560-a916-a58e106c6e8e
health HEALTH_OK
monmap e3: 3 mons at {node02=
192.168.17.212:6789/0,node03=192.168.17.213:6789/0,node04=192.168.17.214:6789/0
}
election epoch 8, quorum 0,1,2 node02,node03,node04
mdsmap e74: 1/1/1 up {0=node02=up:active}
osdmap e324: 14 osds: 14 up, 14 in
pgmap v155699: 768 pgs, 3 pools, 15285 MB data, 1772 objects
91283 MB used, 7700 GB / 7817 GB avail
768 active+clean
* client io 2911 MB/s rd, 90 op/s*
If I do ceph -w, it shows me that it is constantly doing reads, but I have
no idea from where and when it would stop?
I had to remove my CephFS file system and the associated pools and start
things from scratch.
1. Any idea what is happening?
2. When this happens, do you know a better way to get out of the situation
without destroying the filesystem and the pools?
Thanks,
Hadi
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph with SSD and HDD mixed
Hi Johannes, Thanks for your reply. I am naive for this, no idea how to make a configurations or where I can starts? based on the 4 options mentioned. Hope you can expound it further if possible. Best regards, Mario On Tue, Jul 21, 2015 at 2:44 PM, Johannes Formann mlm...@formann.de wrote: Hi, Can someone give an insights, if it possible to mixed SSD with HDD? on the OSD. you’ll have more or less four options: - SSDs for the journals of the OSD-process (SSD must be able to perform good on synchronous writes) - an SSD only pool for „high performance“ data - Using SSDs for the primary copy (fast reads), can be combined with the first - Using a cache pool with an SSD-only pool in front of the main disk-pool How can we speed up the uploading for file for example, as per experience it took around 18mins to load 20Gb images (via glance), in 1Gb network. Or it is just normal? That’s about 20MB/s, for (I guess) sequential writes on a disk only cluster that’s ok. But you can improve that with SSDs, but you have to choose the best option for your setup, depending on the expected workload. greetings Johannes ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph with SSD and HDD mixed
Hi, Mario! Can you give more information about your cluster? Number of nodes, OSDs per node, HDD models etc? In general, you can use SSD as OSD journals ( http://irq0.org/articles/ceph/journal ), but it will give you performance boost on relatively small bursty workloads. If you'll just mix HDD and SSD in one pool it gives you nothing cause clients get ack when every single OSDs with involved PGs returned write ack only so in overal it means the slowest OSD drive timings. May be you can use SSD pool as a cache tier ( http://ceph.com/docs/master/rados/operations/cache-tiering/ ) but I've never tried it and read a number of complaines about it. I think somebody can comment it in a more informative way. Best regards, Vladimir 2015-07-21 6:08 GMT+05:00 Mario Codeniera mario.codeni...@gmail.com: Hi, Can someone give an insights, if it possible to mixed SSD with HDD? on the OSD. How can we speed up the uploading for file for example, as per experience it took around 18mins to load 20Gb images (via glance), in 1Gb network. Or it is just normal? Regards, Mario ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] different omap format in one cluster (.sst + .ldb) - new installed OSD-node don't start any OSD
Hi all, we had an ceph cluster with 7 OSD-nodes (Debian Jessie (because patched tcmalloc) with ceph 0.94) which we expand with one further node. For this node we use puppet with Debian 7.8, because ceph 0.92.2 doesn't install on Jessie (upgrade 0.94.1 work on the other nodes but 0.94.2 looks not clean because the package ceph are still 0.94.1). The ceph.conf is systemwide the same and the OSDs are on all nodes initialized with ceph-deploy (only some exceptions). All OSDs are used ext4, switched from xfs during the cluster run ceph 0.80.7, filestore xattr use omap = true are used in ceph.conf. I'm wondering that the omap-format is different on the nodes. The new wheezy node use .sst files: ls -lsa /var/lib/ceph/osd/ceph-92/current/omap/ ... 2084 -rw-r--r-- 1 root root 2131113 Jul 20 17:45 98.sst 2084 -rw-r--r-- 1 root root 2131913 Jul 20 17:45 99.sst 2084 -rw-r--r-- 1 root root 2130623 Jul 20 17:45 000111.sst ... Due the jessie nodes use levelDB: ls -lsa /var/lib/ceph/osd/ceph-1/current/omap/ ... 2084 -rw-r--r-- 1 root root 2130468 Jul 20 22:33 80.ldb 2084 -rw-r--r-- 1 root root 2130827 Jul 20 22:33 81.ldb 2084 -rw-r--r-- 1 root root 2130171 Jul 20 22:33 88.ldb ... On some OSDs I found old .sst files which came out of wheezy/ceph 0.87 times: ls -lsa /var/lib/ceph/osd/ceph-23/current/omap/*.sst 2096 -rw-r--r-- 1 root root 2142558 Apr 3 15:59 /var/lib/ceph/osd/ceph-23/current/omap/016722.sst 2092 -rw-r--r-- 1 root root 2141968 Apr 3 15:59 /var/lib/ceph/osd/ceph-23/current/omap/016723.sst 2092 -rw-r--r-- 1 root root 2141679 Apr 3 15:59 /var/lib/ceph/osd/ceph-23/current/omap/016724.sst 2096 -rw-r--r-- 1 root root 2142376 Apr 3 15:59 /var/lib/ceph/osd/ceph-23/current/omap/016725.sst 2096 -rw-r--r-- 1 root root 2142227 Apr 3 15:59 /var/lib/ceph/osd/ceph-23/current/omap/016726.sst 2092 -rw-r--r-- 1 root root 2141369 Apr 20 21:23 /var/lib/ceph/osd/ceph-23/current/omap/019470.sst But much more .ldb-files ls -lsa /var/lib/ceph/osd/ceph-23/current/omap/*.ldb | wc -l 128 The config shows for OSDs on both nodes (old and new with .sst-files) as backend leveldb: ceph --admin-daemon /var/run/ceph/ceph-osd.92.asok config show | grep -i omap filestore_omap_backend: leveldb, filestore_debug_omap_check: false, filestore_omap_header_cache_size: 1024, Normaly I would not care about that, but I tried to switch the first OSD-Node to an clean puppet install and see, that none OSD are started. The error message looks a little bit like http://tracker.ceph.com/issues/11429 but this should not happens, because the puppet install has ceph 0.94.2. Error message during start: cat ceph-osd.0.log 2015-07-20 16:51:29.435081 7fb47b126840 0 ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3), process ceph-osd, pid 9803 2015-07-20 16:51:29.457776 7fb47b126840 0 filestore(/var/lib/ceph/osd/ceph-0) backend generic (magic 0xef53) 2015-07-20 16:51:29.460470 7fb47b126840 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is supported and appears to work 2015-07-20 16:51:29.460479 7fb47b126840 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-07-20 16:51:29.485120 7fb47b126840 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: syscall(SYS_syncfs, fd) fully supported 2015-07-20 16:51:29.572670 7fb47b126840 0 filestore(/var/lib/ceph/osd/ceph-0) limited size xattrs 2015-07-20 16:51:29.889599 7fb47b126840 0 filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2015-07-20 16:51:31.517179 7fb47b126840 0 cls cls/hello/cls_hello.cc:271: loading cls_hello 2015-07-20 16:51:31.552366 7fb47b126840 0 osd.0 151644 crush map has features 2303210029056, adjusting msgr requires for clients 2015-07-20 16:51:31.552375 7fb47b126840 0 osd.0 151644 crush map has features 2578087936000 was 8705, adjusting msgr requires for mons 2015-07-20 16:51:31.552382 7fb47b126840 0 osd.0 151644 crush map has features 2578087936000, adjusting msgr requires for osds 2015-07-20 16:51:31.552394 7fb47b126840 0 osd.0 151644 load_pgs 2015-07-20 16:51:42.682678 7fb47b126840 -1 osd/PG.cc: In function 'static epoch_t PG::peek_map_epoch(ObjectStore*, spg_t, ceph::bufferlist*)' thread 7fb47b126840 time 2015-07-20 16:51:42.680036 osd/PG.cc: 2825: FAILED assert(values.size() == 2) ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x72) [0xcdb572] 2: (PG::peek_map_epoch(ObjectStore*, spg_t, ceph::buffer::list*)+0x7b2) [0x908742] 3: (OSD::load_pgs()+0x734) [0x7e9064] 4: (OSD::init()+0xdac) [0x7ed8fc] 5: (main()+0x253e) [0x79069e] 6: (__libc_start_main()+0xfd) [0x7fb47898fead] 7: /usr/bin/ceph-osd() [0x7966b9] NOTE: a copy of the executable, or `objdump -rdS executable` is needed to interpret this. ... Normaly I would
Re: [ceph-users] Firefly 0.80.10 ready to upgrade to?
On Tue, 21 Jul 2015, Olivier Bonvalet wrote: Le lundi 13 juillet 2015 à 11:31 +0100, Gregory Farnum a écrit : On Mon, Jul 13, 2015 at 11:25 AM, Kostis Fardelas dante1...@gmail.com wrote: Hello, it seems that new packages for firefly have been uploaded to repo. However, I can't find any details in Ceph Release notes. There is only one thread in ceph-devel [1], but it is not clear what this new version is about. Is it safe to upgrade from 0.80.9 to 0.80.10? These packages got created and uploaded to the repository without release notes. I'm not sure why but I believe they're safe to use. Hopefully Sage and our release guys can resolve that soon as we've gotten several queries on the subject. :) -Greg ___ Hi, any update on that point ? Packages were uploaded on repositories one month ago. I would appreciate a confirmation go! or NO go! ;) Sorry, I was sick and this dropped off my list. I'll post the release notes today. Thanks! sage ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Firefly 0.80.10 ready to upgrade to?
Le lundi 13 juillet 2015 à 11:31 +0100, Gregory Farnum a écrit : On Mon, Jul 13, 2015 at 11:25 AM, Kostis Fardelas dante1...@gmail.com wrote: Hello, it seems that new packages for firefly have been uploaded to repo. However, I can't find any details in Ceph Release notes. There is only one thread in ceph-devel [1], but it is not clear what this new version is about. Is it safe to upgrade from 0.80.9 to 0.80.10? These packages got created and uploaded to the repository without release notes. I'm not sure why but I believe they're safe to use. Hopefully Sage and our release guys can resolve that soon as we've gotten several queries on the subject. :) -Greg ___ Hi, any update on that point ? Packages were uploaded on repositories one month ago. I would appreciate a confirmation go! or NO go! ;) thanks, Olivier ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Firefly 0.80.10 ready to upgrade to?
Le mardi 21 juillet 2015 à 07:06 -0700, Sage Weil a écrit : On Tue, 21 Jul 2015, Olivier Bonvalet wrote: Le lundi 13 juillet 2015 à 11:31 +0100, Gregory Farnum a écrit : On Mon, Jul 13, 2015 at 11:25 AM, Kostis Fardelas dante1...@gmail.com wrote: Hello, it seems that new packages for firefly have been uploaded to repo. However, I can't find any details in Ceph Release notes. There is only one thread in ceph-devel [1], but it is not clear what this new version is about. Is it safe to upgrade from 0.80.9 to 0.80.10? These packages got created and uploaded to the repository without release notes. I'm not sure why but I believe they're safe to use. Hopefully Sage and our release guys can resolve that soon as we've gotten several queries on the subject. :) -Greg ___ Hi, any update on that point ? Packages were uploaded on repositories one month ago. I would appreciate a confirmation go! or NO go! ;) Sorry, I was sick and this dropped off my list. I'll post the release notes today. Thanks! sage Great, I take that for a go!. Thanks Sage :) ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
