Dear ceph-users,
I'm preparing a Ceph cluster on my debian machines. I successfully walked
true the Preflight installation guide on the website.
Now i'm stuck at the STORAGE CLUSTER QUICK START, when i enter the
following command:
ceph-deploy mon create-initial
I get the following errors:
[ce
On Thu, Sep 22, 2016 at 1:42 AM, Chris Jones wrote:
> Ceph-rust for librados has been released. It's an API interface in Rust for
> all of librados that's a thin layer above the C APIs. There are low-level
> direct access and higher level Rust helpers that make working directly with
> librados sim
Any guidance on this? I have osd_snap_trim_sleep set to 1 and it seems to have
tempered some of the issues but its still bad enough that NFS storage off RBD
volumes become unavailable for over 3 minutes.
It seems that the activity which the snapshot deletes are actioned triggers
massive disk
So just to put more info out there, here is what I’m seeing with a Spark/HDFS
client:
2016-09-21 20:09:25.076595 7fd61c16f700 0 -- 192.168.1.157:0/634334964 >>
192.168.1.190:6802/32183 pipe(0x7fd5fcef8ca0 sd=66 :53864 s=2 pgs=50445 cs=1
l=0 c=0x7fd5fdd371d0).fault, initiating reconnect
2016-09
Felix,
According to my tests there is difference in performance between usual named
buckets (test, test01, test02), uuid-named buckets (like
'7c9e4a81-df86-4c9d-a681-3a570de109db') or just date ('2016-09-20-16h').
Getting ~3x more upload performance (220 uploads\s vs 650 uploads\s) with
SSD-bac
I also went and bumped mds_cache_size up to 1 million… still seeing cache
pressure, but I might just need to evict those clients…
On 9/21/16, 9:24 PM, "Heller, Chris" wrote:
What is the interesting value in ‘session ls’? Is it ‘num_leases’ or
‘num_caps’ leases appears to be, on average, 1.
What is the interesting value in ‘session ls’? Is it ‘num_leases’ or ‘num_caps’
leases appears to be, on average, 1. But caps seems to be 16385 for many many
clients!
-Chris
On 9/21/16, 9:22 PM, "Gregory Farnum" wrote:
On Wed, Sep 21, 2016 at 6:13 PM, Heller, Chris wrote:
> I’m suspe
Ben,
Works fine as far as I see:
[root@273aa9f2ee9f /]# s3cmd mb s3://test
Bucket 's3://test/' created
[root@273aa9f2ee9f /]# s3cmd put /etc/hosts s3://test
upload: '/etc/hosts' -> 's3://test/hosts' [1 of 1]
196 of 196 100% in0s 404.87 B/s done
[root@273aa9f2ee9f /]# s3cmd ls s3://te
On Wed, Sep 21, 2016 at 6:13 PM, Heller, Chris wrote:
> I’m suspecting something similar, we have millions of files and can read a
> huge subset of them at a time, presently the client is Spark 1.5.2 which I
> suspect is leaving the closing of file descriptors up to the garbage
> collector. Tha
I’m suspecting something similar, we have millions of files and can read a huge
subset of them at a time, presently the client is Spark 1.5.2 which I suspect
is leaving the closing of file descriptors up to the garbage collector. That
said, I’d like to know if I could verify this theory using th
Thanks. Will try it out once we get on Jewel.
Just curious, does bucket deletion with --purge-objects work via
radosgw-admin with the no index option?
If not, i imagine rados could be used to delete them manually by prefix.
On Sep 21, 2016 6:02 PM, "Stas Starikevich"
wrote:
> Hi Ben,
>
> Since
Hi Ben,
Since the 'Jewel' RadosGW supports blind buckets.
To enable blind buckets configuration I used:
radosgw-admin zone get --rgw-zone=default > default-zone.json
#change index_type from 0 to 1
vi default-zone.json
radosgw-admin zone set --rgw-zone=default --infile default-zone.json
To apply
While attempting to upgrade a 1200+ OSD cluster from 0.94.6 to 0.94.9 I've
run into serious performance issues every time I restart an OSD.
At first I thought the problem I was running into was caused by the osdmap
encoding bug that Dan and Wido ran into when upgrading to 0.94.7, because
I was see
Hello,
We have 2 Ceph clusters running in two separate data centers, each one with
3 mons, 3 rgws, and 5 osds. I am attempting to get bi-directional
multi-site replication setup as described in the ceph documentation here:
http://docs.ceph.com/docs/jewel/radosgw/multisite/
We are running Jewel v
Unfortunately, it sounds like the image's header object was lost
during your corruption event. While you can manually retrieve the
image data blocks from the cluster, undoubtedly many might be lost
and/or corrupted as well.
You'll first need to determine the internal id of your image:
$ rados --po
On Wed, Sep 21, 2016 at 1:16 PM, Heller, Chris wrote:
> Ok. I just ran into this issue again. The mds rolled after many clients were
> failing to relieve cache pressure.
That definitely could have had something to do with it, if say they
overloaded the MDS so much it got stuck in a directory rea
Ok. I just ran into this issue again. The mds rolled after many clients were
failing to relieve cache pressure.
Now here is the result of `ceph –s`
# ceph -s
cluster b126570e-9e7c-0bb2-991f-ecf9abe3afa0
health HEALTH_OK
monmap e1: 5 mons at
{a154=192.168.1.154:6789/0,a155=192.168.
On 21 September 2016 at 17:28, Haomai Wang wrote:
> BTW, why you need to iterate so much objects. I think it should be
> done by other ways to achieve the goal.
>
Mostly it's just a brute force way to identify objects that shouldn't
exist, or objects that have been orphaned (e.g: last modific
Nice, thanks! Must have missed that one. It might work well for our use
case since we don't really need the index.
-Ben
On Wed, Sep 21, 2016 at 11:23 AM, Gregory Farnum wrote:
> On Wednesday, September 21, 2016, Ben Hines wrote:
>
>> Yes, 200 million is way too big for a single ceph RGW bucket
On Wednesday, September 21, 2016, Ben Hines wrote:
> Yes, 200 million is way too big for a single ceph RGW bucket. We
> encountered this problem early on and sharded our buckets into 20 buckets,
> each which have the sharded bucket index with 20 shards.
>
> Unfortunately, enabling the sharded RGW
Yes, 200 million is way too big for a single ceph RGW bucket. We
encountered this problem early on and sharded our buckets into 20 buckets,
each which have the sharded bucket index with 20 shards.
Unfortunately, enabling the sharded RGW index requires recreating the
bucket and all objects.
The fa
It seems the trigger for the problem is this:
> 24.9141130d 1000527. [write 0~242] snapc 1=[] ondisk+write
> e320)
>-40> 2016-09-20 20:38:02.007942 708f67bbd700 0
> filestore(/var/lib/ceph/osd/ceph-0) write couldn't open
> 24.32_head/#24:4d11884b:::1000504.:head#: (24)
Hi,
Regarding to Amazon S3 documentation, it is advised to insert a bit of
random chars in the bucket name in order to gain performance. This is
related to how Amazon store key names. It looks like they store an index of
object key names in each region.
http://docs.aws.amazon.com/AmazonS3/latest/
Ceph-rust for librados has been released. It's an API interface in Rust for
all of librados that's a thin layer above the C APIs. There are low-level
direct access and higher level Rust helpers that make working directly with
librados simple.
The official repo is:
https://github.com/ceph/ceph-rus
> Op 21 september 2016 om 17:23 schreef Iain Buclaw :
>
>
> On 20 September 2016 at 19:27, Gregory Farnum wrote:
> > In librados getting a stat is basically equivalent to reading a small
> > object; there's not an index or anything so FileStore needs to descend its
> > folder hierarchy. If look
BTW, why you need to iterate so much objects. I think it should be
done by other ways to achieve the goal.
On Wed, Sep 21, 2016 at 11:23 PM, Iain Buclaw wrote:
> On 20 September 2016 at 19:27, Gregory Farnum wrote:
>> In librados getting a stat is basically equivalent to reading a small
>> o
On 20 September 2016 at 19:27, Gregory Farnum wrote:
> In librados getting a stat is basically equivalent to reading a small
> object; there's not an index or anything so FileStore needs to descend its
> folder hierarchy. If looking at metadata for all the objects in the system
> efficiently is im
Perhaps related, I was watching the active mds with debug_mds set to 5/5, when
I saw this in the log:
2016-09-21 15:13:26.067698 7fbaec248700 0 -- 192.168.1.196:6802/13581 >>
192.168.1.238:0/3488321578 pipe(0x55db000 sd=49 :6802 s=2 pgs=2 cs=1 l=0
c=0x5631ce0).fault with nothing to send, going
Hello,
I have a Ceph Jewel cluster with 4 osds and only one monitor integrated
with Openstack Mitaka.
Two OSD were down, with a service restart one of them was recovered. The
cluster began to recover and was OK. Finally the disk of the other OSD was
corrupted and the solution was a format and rec
On 21 September 2016 at 02:57, Haomai Wang wrote:
> On Wed, Sep 21, 2016 at 2:41 AM, Wido den Hollander wrote:
>>
>>> Op 20 september 2016 om 20:30 schreef Haomai Wang :
>>>
>>>
>>> On Wed, Sep 21, 2016 at 2:26 AM, Wido den Hollander wrote:
>>> >
>>> >> Op 20 september 2016 om 19:27 schreef Greg
Dear ceph-users,
I'm preparing a Ceph cluster on my debian machines. I successfully walked
true the Preflight installation guide on the website.
Now i'm stuck at the STORAGE CLUSTER QUICK START, when i enter the
following command:
ceph-deploy mon create-initial
I get the following errors:
[ce
I’ll see if I can capture the output the next time this issue arises, but in
general the output looks as if nothing is wrong. No OSD are down, a ‘ceph
health detail’ results in HEALTH_OK, the mds server is in the up:active state,
in general it’s as if nothing is wrong server side (at least from
On Wed, Sep 21, 2016 at 6:30 AM, Heller, Chris wrote:
> I’m running a production 0.94.7 Ceph cluster, and have been seeing a
> periodic issue arise where in all my MDS clients will become stuck, and the
> fix so far has been to restart the active MDS (sometimes I need to restart
> the subsequent a
Hi,
- Original Message -
> From: "yiming xie"
> To: ceph-users@lists.ceph.com
> Sent: Wednesday, September 21, 2016 3:53:35 AM
> Subject: [ceph-users] Help on RGW NFS function
>
> Hi,
> I have some question about rgw nfs.
>
> ceph release notes: You can now access radosgw buckets via NF
I’m running a production 0.94.7 Ceph cluster, and have been seeing a periodic
issue arise where in all my MDS clients will become stuck, and the fix so far
has been to restart the active MDS (sometimes I need to restart the subsequent
active MDS as well).
These clients are using the cephfs-hado
I seriously doubt that it's ever going to be a winning strategy to let
rgw index objects go to a cold tier. Some practical problems:
1) We don't track omap size (the leveldb entries for an object)
because it would turn writes into rmw's -- so they always show up as 0
size. Thus, the target_max_by
Ah, same question then. If we can get logging on the primary for one
of those pgs, it should be fairly obvious.
-Sam
On Wed, Sep 21, 2016 at 4:08 AM, Pavan Rallabhandi
wrote:
> We find this as well in our fresh built Jewel clusters, and seems to happen
> only with a handful of PGs from couple o
Hi,
Responded inline.
On Wed, Sep 21, 2016 at 4:54 AM, Brian Chang-Chien
wrote:
>
>
> [global]
> ...
> debug rgw = 20
> [client.radosgw.gateway]
> host = brianceph
> rgw keystone url = http://10.62.13.253:35357
> rgw keystone admin token = 7bb8e26cbc714c47a26ffec3d96f246f
> rgw keystone accepted
We find this as well in our fresh built Jewel clusters, and seems to happen
only with a handful of PGs from couple of pools.
Thanks!
On 9/21/16, 3:14 PM, "ceph-users on behalf of Tobias Böhm"
wrote:
Hi,
there is an open bug in the tracker: http://tracker.ceph.com/issues/16474
Looks like the OSD didn't like an error return it got from the
underlying fs. Can you reproduce with
debug filestore = 20
debug osd = 20
debug ms = 1
on the osd and post the whole log?
-Sam
On Wed, Sep 21, 2016 at 12:10 AM, Peter Maloney
wrote:
> Hi,
>
> I created a one disk osd with data and
Can you reproduce with logging on the primary for that pg?
debug osd = 20
debug filestore = 20
debug ms = 1
Since restarting the osd may be a workaround, can you inject the debug
values without restarting the daemon?
-Sam
On Wed, Sep 21, 2016 at 2:44 AM, Tobias Böhm wrote:
> Hi,
>
> there is an
Hi Radoslaw Zarzynski
I also test Ceph Hammer 0.94.7 for rados gateway
the result still response 401 , so i post ceph.conf, radosgw.log as
following
ceph.conf
when i call swift cmd , radosgw.log
2016-09-21 17:47:12.141783 7fc7f4bf5700 10
content=MIIBzgYJKoZIhvcNAQcCoIIBvzCCAbsCAQExDTALBglghk
Hi,
there is an open bug in the tracker: http://tracker.ceph.com/issues/16474
It also suggests restarting OSDs as a workaround. We faced the same issue after
increasing the number of PGs in our cluster and restarting OSDs solved it as
well.
Tobias
> Am 21.09.2016 um 11:26 schrieb Dan van der
There was a thread about this a few days ago:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-September/012857.html
And the OP found a workaround.
Looks like a bug though... (by default PGs scrub at most once per day).
-- dan
On Tue, Sep 20, 2016 at 10:43 PM, Martin Bureau wrote:
> He
Hi,
I have some question about rgw nfs.
ceph release notes: You can now access radosgw buckets via NFS (experimental).
In addition to the sentence, ceph documents does not do any explanation
I don't understand the experimental implications.
1. RGW nfs functional integrity of it? If nfs function i
Hi,
There is a cluster of ceph. To accelerate dokinuli ssd into it, the card: I
ssd gathered in a separate pool and a prescribed rule. But I'm getting
the following error msg:
[ceph@ceph-adm test]$ ceph -s
cluster a73f8be5-0fa6-4a82-aa9b-22121246398e
health HEALTH_WARN
64 p
Hi,
I created a one disk osd with data and separate journal on the same lvm
volume group just for test, one mon, one mds on my desktop.
I managed to crash the osd just by mounting cephfs and doing cp -a of
the linux-stable git tree into it. It crashed after copying 2.1G which
only covers some of
47 matches
Mail list logo
