Yeah, the number of shards is configurable ("rgw usage num shards"? or
something).
Are you sure you aren't using it? This feature is not enabled by
default, someone had to explicitly set "rgw enable usage log" for you
to run into this problem.
Paul
--
Paul Emmerich
Looking for help with your
Paul,
Apparently never. Appears to (potentially) have every request from the
beginning of time (late last year, in my case). In our use case, we
don't really need this data (not multi-tenant), so I might simply clear it.
But in the case where this were an extremely high transaction cluster
Looks like the usage log (radosgw-admin usage show), how often do you trim it?
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Mon, Oct 14, 2019 at 11:55 PM Troy Ablan
Hi folks,
Mimic cluster here, RGW pool with only default zone. I have a
persistent error here
LARGE_OMAP_OBJECTS 1 large omap objects
1 large objects found in pool 'default.rgw.log'
Search the cluster log for 'Large omap object found' for more
details.
I think I've narrowed it
How big is the mon's DB? As in just the total size of the directory you copied
FWIW I recently had to perform mon surgery on a 14.2.4 (or was it
14.2.2?) cluster with 8 GB mon size and I encountered no such problems
while syncing a new mon which took 10 minutes or so.
Paul
--
Paul Emmerich
On Mon, Oct 14, 2019 at 04:31:22PM +0200, Nikola Ciprich wrote:
> On Mon, Oct 14, 2019 at 01:40:19PM +0200, Harald Staub wrote:
> > Probably same problem here. When I try to add another MON, "ceph
> > health" becomes mostly unresponsive. One of the existing ceph-mon
> > processes uses 100% CPU for
Hi Igor,
It is the old Jewel (v10.2.11). This case is happen after I
do cache-try-flush-evict-all or cache-flush-evict-all on respected tier
pool.
Best regards,
On Mon, Oct 14, 2019 at 7:38 PM Igor Fedotov wrote:
> Hi Lazuardi,
>
> never seen that. Just wondering what Ceph version are you
On Mon, Oct 14, 2019 at 01:40:19PM +0200, Harald Staub wrote:
> Probably same problem here. When I try to add another MON, "ceph
> health" becomes mostly unresponsive. One of the existing ceph-mon
> processes uses 100% CPU for several minutes. Tried it on 2 test
> clusters (14.2.4, 3 MONs, 5
On Thu, Oct 10, 2019 at 2:23 PM huxia...@horebdata.cn
wrote:
>
> Hi, folks,
>
> I have a middle-size Ceph cluster as cinder backup for openstack (queens).
> Duing testing, one Ceph node went down unexpected and powered up again ca 10
> minutes later, Ceph cluster starts PG recovery. To my
@Mike
Did you have the chance to update download.ceph.com repositories for the new
version?
I just tested the packages from shaman in our DEV environment and it seems to
fix the work - after updating the packages i was not able to reproduce the
error again and tcmu-runner starts up without
Hi Lazuardi,
never seen that. Just wondering what Ceph version are you running?
Thanks,
Igor
On 10/8/2019 3:52 PM, Lazuardi Nasution wrote:
Hi,
I get following weird negative objects number on tiering. Why is this
happening? How to get back to normal?
Best regards,
[root@management-a
Hi,
Any body has same problem with my case?
Best regards,
On Tue, Oct 8, 2019, 19:52 Lazuardi Nasution wrote:
> Hi,
>
> I get following weird negative objects number on tiering. Why is this
> happening? How to get back to normal?
>
> Best regards,
>
> [root@management-a ~]# ceph df detail
>
Probably same problem here. When I try to add another MON, "ceph health"
becomes mostly unresponsive. One of the existing ceph-mon processes uses
100% CPU for several minutes. Tried it on 2 test clusters (14.2.4, 3
MONs, 5 storage nodes with around 2 hdd osds each). To avoid errors like
"lease
Hi Ernesto,
I just opened the Dashboard and there is no menu at the top-right. Also no "?".
I have a menu at the top-left which has the following items: Cluster health,
Cluster, Block and Filesystems.
Running Ceph version 12.2.8-89.
Kind regards,
Sinan Polat
> Op 11 oktober 2019 om 22:09
dear ceph users and developers,
on one of our production clusters, we got into pretty unpleasant situation.
After rebooting one of the nodes, when trying to start monitor, whole cluster
seems to hang, including IO, ceph -s etc. When this mon is stopped again,
everything seems to continue.
On Sat, Oct 12, 2019 at 1:10 AM Kenneth Waegeman
wrote:
> Hi all,
>
> After solving some pg inconsistency problems, my fs is still in
> trouble. my mds's are crashing with this error:
>
>
> > -5> 2019-10-11 19:02:55.375 7f2d39f10700 1 mds.1.564276 rejoin_start
> > -4> 2019-10-11
I have created an anonymized crash log at
https://pastebin.ubuntu.com/p/YsVXQQTBCM/ in the hopes that it can help someone
understand what's leading to our MDS outage.
Thanks in advance for any assistance.
From: Pickett, Neale T
Sent: Thursday, October 10,
Hi Ernesto,
Thanks for the information! I didn’t know about the existence of the REST
Dashboard API. I will check that out, Thanks again!
Sinan
> Op 11 okt. 2019 om 21:06 heeft Ernesto Puerta het
> volgende geschreven:
>
> Hi Sinan,
>
> If it's in the Dashboard, it sure comes from the
Hi Sinan,
If it's in the Dashboard, it sure comes from the Dashboard REST API (which
is an API completely unrelated to the RESTful Module).
To check the Dashboard REST API, log in there and click on the top-right
"?" menu, and in the dropdown, click on "API". That will lead you to the
Hi all,
After solving some pg inconsistency problems, my fs is still in
trouble. my mds's are crashing with this error:
-5> 2019-10-11 19:02:55.375 7f2d39f10700 1 mds.1.564276 rejoin_start
-4> 2019-10-11 19:02:55.385 7f2d3d717700 5 mds.beacon.mds01
received beacon reply
On 11/10/2019 01:21, Brad Hubbard wrote:
On Fri, Oct 11, 2019 at 12:27 AM Kenneth Waegeman
wrote:
Hi Brad, all,
Pool 6 has min_size 2:
pool 6 'metadata' replicated size 3 min_size 2 crush_rule 1 object_hash
rjenkins pg_num 1024 pgp_num 1024 autoscale_mode warn last_change 172476
flags
> Am 11.10.2019 um 14:07 schrieb Igor Fedotov :
>
>
> Hi!
>
> originally your issue looked like the ones from
> https://tracker.ceph.com/issues/42223
>
> And it looks like lack of some key information for FreeListManager in RocksDB.
>
> Once you have it present we can check the content of
Hi!
originally your issue looked like the ones from
https://tracker.ceph.com/issues/42223
And it looks like lack of some key information for FreeListManager in
RocksDB.
Once you have it present we can check the content of the RocksDB to
prove this hypothesis, please let me know if you
I have setup the realm, zonegroup and master zone. Now Iam pulling the
realm details from master to secondary...it failed with "request failed:
(22) invalid argument"
==
radosgw-admin realm pull --url={url}, --access-key={key} --secret={key}
request failed: (22) Invalid argument
==
On Mon, Oct
Hello, ceph-users.
Our mds servers keep segfaulting from a failed assertion, and for the first
time I can't find anyone else who's posted about this problem. None of them are
able to stay up, so our cephfs is down.
We recently had to truncate the journal log after an upgrade to nautilus, and
Currently I am getting the pool statistics (especially USED/MAX AVAIL) via the
command line:
ceph df -f json-pretty| jq '.pools[] | select(.name == "poolname") |
.stats.max_avail'
ceph df -f json-pretty| jq '.pools[] | select(.name == "poolname") |
.stats.bytes_used'
Command "ceph df" does not
On Fri, Oct 11, 2019 at 12:27 AM Kenneth Waegeman
wrote:
>
> Hi Brad, all,
>
> Pool 6 has min_size 2:
>
> pool 6 'metadata' replicated size 3 min_size 2 crush_rule 1 object_hash
> rjenkins pg_num 1024 pgp_num 1024 autoscale_mode warn last_change 172476
> flags hashpspool stripe_width 0
Hi, folks,
I have a middle-size Ceph cluster as cinder backup for openstack (queens).
Duing testing, one Ceph node went down unexpected and powered up again ca 10
minutes later, Ceph cluster starts PG recovery. To my surprise, VM IOPS drops
dramatically during Ceph recovery, from ca. 13K IOPS
Hi Brad, all,
Pool 6 has min_size 2:
pool 6 'metadata' replicated size 3 min_size 2 crush_rule 1 object_hash
rjenkins pg_num 1024 pgp_num 1024 autoscale_mode warn last_change 172476
flags hashpspool stripe_width 0 application cephfs
The output for all the inconsistent pgs is this:
{
I've also encountered this issue on a cluster yesterday; one CPU got
stuck in an infinite loop in get_obj_data::flush and it stopped
serving requests. I've updated the tracker issue accordingly.
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
Hi,
Currently I am getting the pool statistics (especially USED/MAX AVAIL) via the
command line:
ceph df -f json-pretty| jq '.pools[] | select(.name == "poolname") |
.stats.max_avail'
ceph df -f json-pretty| jq '.pools[] | select(.name == "poolname") |
.stats.bytes_used'
Command "ceph df" does
After trying to disable the paxos service trim temporarily (since that seemed
to trigger it initially), we now see this:
"assert_condition": "from != to",
"assert_func": "void PaxosService::trim(MonitorDBStore::TransactionRef,
version_t, version_t)",
"assert_file":
Thanks, Patrick. Looks like the fix is awaiting review, I guess my options
are to hold tight for 14.2.5 or patch myself if I get desperate. I've seen
this crash about 4 times over the past 96 hours, is there anything I can do
to mitigate the issue in the meantime?
On Wed, Oct 9, 2019 at 9:23 PM
Does pool 6 have min_size = 1 set?
https://tracker.ceph.com/issues/24994#note-5 would possibly be helpful
here, depending on what the output of the following command looks
like.
# rados list-inconsistent-obj [pgid] --format=json-pretty
On Thu, Oct 10, 2019 at 8:16 PM Kenneth Waegeman
wrote:
>
Hi all,
After some node failure and rebalancing, we have a lot of pg's in
inconsistent state. I tried to repair, but it din't work. This is also
in the logs:
2019-10-10 11:23:27.221 7ff54c9b0700 0 log_channel(cluster) log [DBG]
: 6.327 repair starts
2019-10-10 11:23:27.431 7ff5509b8700 -1
How do I Import an osdmap in Nautilus? I saw documentation for older version
but it seems one now can only export but not import anymore?
Am Donnerstag, 10. Oktober 2019, 08:52:03 OESZ hat Philippe D'Anjou
Folgendes geschrieben:
I dont think this has anything to do with CephFS, the
I dont think this has anything to do with CephFS, the mon crashes for same
reason even without the mds running.I have still the old rocksdb files but they
had a corruption issue, not sure if that's easier to fix, there havent been any
changes on the cluster in between.
This is a disaster
On Wed, Oct 9, 2019 at 10:58 AM Vladimir Brik <
vladimir.b...@icecube.wisc.edu> wrote:
> Best I can tell, automatic cache sizing is enabled and all related
> settings are at their default values.
>
> Looking through cache tunables, I came across
> osd_memory_expected_fragmentation, which the docs
Awesome! Sorry it took so long.
On Thu, Oct 10, 2019 at 12:44 AM Marc Roos wrote:
>
>
> Brad, many thanks!!! My cluster has finally HEALTH_OK af 1,5 year or so!
> :)
>
>
> -Original Message-
> Subject: Re: Ceph pg repair clone_missing?
>
> On Fri, Oct 4, 2019 at 6:09 PM Marc Roos
>
Looks like this bug: https://tracker.ceph.com/issues/41148
On Wed, Oct 9, 2019 at 1:15 PM David C wrote:
>
> Hi Daniel
>
> Thanks for looking into this. I hadn't installed ceph-debuginfo, here's the
> bt with line numbers:
>
> #0 operator uint64_t (this=0x10) at
>
Hi Daniel
Thanks for looking into this. I hadn't installed ceph-debuginfo, here's the
bt with line numbers:
#0 operator uint64_t (this=0x10) at
/usr/src/debug/ceph-14.2.2/src/include/object.h:123
#1 Client::fill_statx (this=this@entry=0x274b980, in=0x0, mask=mask@entry=341,
Best I can tell, automatic cache sizing is enabled and all related
settings are at their default values.
Looking through cache tunables, I came across
osd_memory_expected_fragmentation, which the docs define as "estimate
the percent of memory fragmentation". What's the formula to compute
On Mon, Oct 7, 2019 at 11:11 PM Philippe D'Anjou
wrote:
>
> Hi,
> unfortunately it's single mon, because we had major outage on this cluster
> and it's just being used to copy off data now. We werent able to add more
> mons because once a second mon was added it crashed the first one (there's a
Thanks - yeah jewel is old But i meant to say nautilus and not luminous.
The first option probably wont work for me. Since both sides are active and the
application1 needs to write in both places as http://application1.something.com.
The 2nd one in theory should work. I'm using haproxy and it
On Mon, Oct 7, 2019 at 7:20 AM Vladimir Brik
wrote:
>
> > Do you have statistics on the size of the OSDMaps or count of them
> > which were being maintained by the OSDs?
> No, I don't think so. How can I find this information?
Hmm I don't know if we directly expose the size of maps. There are
Boy, Jewel is pretty old. Even Luminous is getting up there. There have been a
lot of multisite improvements in Mimic and now Nautilus, so you might want to
consider upgrading all the way to 14.2.4.
Anyway, the way we solve this is by giving each zone a different name (eg
Hello,
I have a question about multi site configuration. I have 2 clusters configured
in a single realm and zonegroup. One cluster is the master zone and the other
the slave. Lets assume the first cluster can be reached at
http://application1.something.com and the 2nd one is
Brad, many thanks!!! My cluster has finally HEALTH_OK af 1,5 year or so!
:)
-Original Message-
Subject: Re: Ceph pg repair clone_missing?
On Fri, Oct 4, 2019 at 6:09 PM Marc Roos
wrote:
>
> >
> >Try something like the following on each OSD that holds a copy of
>
On Fri, Oct 4, 2019 at 6:09 PM Marc Roos wrote:
>
> >
> >Try something like the following on each OSD that holds a copy of
> >rbd_data.1f114174b0dc51.0974 and see what output you get.
> >Note that you can drop the bluestore flag if they are not bluestore
> >osds and you will need
Hello.
I've created an rgw installation, had uploaded about 60M files into a
single bucket. Removal had looked as a long adventure, so I "ceph osd
pool rm'ed" both default.rgw.data and default.rgw.index.
Now I have this:
# rados lspools
.rgw.root
default.rgw.control
default.rgw.meta
Hi,
I get following weird negative objects number on tiering. Why is this
happening? How to get back to normal?
Best regards,
[root@management-a ~]# ceph df detail
GLOBAL:
SIZE AVAIL RAW USED %RAW USED OBJECTS
446T 184T 261T 58.62 22092k
POOLS:
Hi,
the default for this warning changed recently (see other similar
threads on the mailing list), it was 2 million before 14.2.3.
I don't think the new default of 200k is a good choice, so increasing
it is a reasonable work-around.
Paul
--
Paul Emmerich
Looking for help with your Ceph
Hi,
there is also /var/log/ceph/ceph.log on the MONs, it has the stats
you're asking for. Does this answer your question?
Regards,
Eugen
Zitat von nokia ceph :
Hi Team,
With default log settings , the ceph stats will be logged like
cluster [INF] pgmap v30410386: 8192 pgs: 8192
Hi Team,
With default log settings , the ceph stats will be logged like
cluster [INF] pgmap v30410386: 8192 pgs: 8192 active+clean; 445 TB data,
1339 TB used, 852 TB / 2191 TB avail; 188 kB/s rd, 217 MB/s wr, 1618 op/s
Jewel : on mon logs
Nautilus : on mgr logs
Luminous : not able to view
Hi,unfortunately it's single mon, because we had major outage on this cluster
and it's just being used to copy off data now. We werent able to add more mons
because once a second mon was added it crashed the first one (there's a bug
tracker ticket).
I still have old rocksdb files before I ran
On Sun, Oct 6, 2019 at 1:08 AM Philippe D'Anjou
wrote:
>
> I had to use rocksdb repair tool before because the rocksdb files got
> corrupted, for another reason (another bug possibly). Maybe that is why now
> it crash loops, although it ran fine for a day.
Yeah looks like it lost a bit of
Client::fill_statx() is a fairly large function, so it's hard to know
what's causing the crash. Can you get line numbers from your backtrace?
Daniel
On 10/7/19 9:59 AM, David C wrote:
Hi All
Further to my previous messages, I upgraded
to libcephfs2-14.2.2-0.el7.x86_64 as suggested and
> Do you have statistics on the size of the OSDMaps or count of them
> which were being maintained by the OSDs?
No, I don't think so. How can I find this information?
Memory consumption started to climb again:
https://icecube.wisc.edu/~vbrik/graph-3.png
Some more info (not sure if relevant or
Hi All
Further to my previous messages, I upgraded
to libcephfs2-14.2.2-0.el7.x86_64 as suggested and things certainly seem a
lot more stable, I have had some crashes though, could someone assist in
debugging this latest crash please?
(gdb) bt
#0 0x7fce4e9fc1bb in Client::fill_statx(Inode*,
Thank you...Let me confirm the same..and update here.
On Sat, Oct 5, 2019 at 12:27 AM wrote:
> Swami;
>
> For 12.2.11 (Luminous), the previously linked document would be:
>
> https://docs.ceph.com/docs/luminous/radosgw/multisite/#migrating-a-single-site-system-to-multi-site
>
> Thank you,
>
>
I've adjusted the threshold:
ceph config set osd osd_deep_scrub_large_omap_object_key_threshold 35
Colleague suggested that this will take effect on the next deep-scrub.
Is the default of 200,000 too small? will this be adjusted in future
releases or is it meant to be adjusted in some
I followed some other suggested steps, and have this:
root@cnx-17:/var/log/ceph# zcat ceph-osd.178.log.?.gz|fgrep Large
2019-10-02 13:28:39.412 7f482ab1c700 0 log_channel(cluster) log [WRN] :
Large omap object found. Object: 2:654134d2:::mds0_openfiles.0:head Key
count: 306331 Size (bytes):
Out of the blue this popped up (on an otherwise healthy cluster):
HEALTH_WARN 1 large omap objects
LARGE_OMAP_OBJECTS 1 large omap objects
1 large objects found in pool 'cephfs_metadata'
Search the cluster log for 'Large omap object found' for more details.
"Search the cluster log" is
Hi,
On inspecting new installed cluster (Nautilus), I find following result.
ssd-test pool is cache pool for hdd-test pool. After running some RBD bench
and delete all rbd images used for benchmarking, it there is some hidden
objects inside both pools (except rbd_directory, rbd_info and
I had to use rocksdb repair tool before because the rocksdb files got
corrupted, for another reason (another bug possibly). Maybe that is why now it
crash loops, although it ran fine for a day.What is meant with "turn it off and
rebuild from remainder"?
Am Samstag, 5. Oktober 2019,
Thanks! I’ll remove my patch from my local build of the 4.19 kernel and
upgrade to 4.19.77. Appreciate the quick fix.
Thanks,
--
Kenneth Van Alstyne
Systems Architect
M: 228.547.8045
15052 Conference Center Dr, Chantilly, VA 20151
perspecta
On Oct 5, 2019, at 7:29 AM, Ilya Dryomov
On Tue, Oct 1, 2019 at 9:12 PM Jeff Layton wrote:
>
> On Tue, 2019-10-01 at 15:04 -0400, Sasha Levin wrote:
> > On Tue, Oct 01, 2019 at 01:54:45PM -0400, Jeff Layton wrote:
> > > On Tue, 2019-10-01 at 19:03 +0200, Ilya Dryomov wrote:
> > > > On Tue, Oct 1, 2019 at 6:41 PM Kenneth Van Alstyne
> >
On Wed, Oct 2, 2019 at 9:48 AM Stefan Kooman wrote:
> According to [1] there are new parameters in place to have the MDS
> behave more stable. Quoting that blog post "One of the more recent
> issues weve discovered is that an MDS with a very large cache (64+GB)
> will hang during certain recovery
Do you have statistics on the size of the OSDMaps or count of them
which were being maintained by the OSDs? I'm not sure why having noout
set would change that if all the nodes were alive, but that's my bet.
-Greg
On Thu, Oct 3, 2019 at 7:04 AM Vladimir Brik
wrote:
>
> And, just as unexpectedly,
Hmm, that assert means the monitor tried to grab an OSDMap it had on
disk but it didn't work. (In particular, a "pinned" full map which we
kept around after trimming the others to save on disk space.)
That *could* be a bug where we didn't have the pinned map and should
have (or incorrectly
The 4M throughput numbers you see now ( 150 MB/s read, 60 MB/s write)
are probably limited by your 1G network, and can probably go higher if
you increase it ( 10G or use active bonds).
In real life, the applications and wokloads determine the block size, io
depths, whether it is sequential
Thank you guys,
I changed FIO parameters and it looks far better now - reading about
150MB/s, writing over 60MB/s
Now, the question is, what could I change in my setup to make it this
fast - the RBD is used as LVM PV for a VG shared between Xen
hypervisors, this is the PV:
--- Physical
Swami;
For 12.2.11 (Luminous), the previously linked document would be:
https://docs.ceph.com/docs/luminous/radosgw/multisite/#migrating-a-single-site-system-to-multi-site
Thank you,
Dominic L. Hilsbos, MBA
Director – Information Technology
Perform Air International Inc.
hi all,
maybe to clarify a bit, e.g.
https://indico.cern.ch/event/755842/contributions/3243386/attachments/1784159/2904041/2019-jcollet-openlab.pdf
clearly shows that the db+wal disks are not saturated,
but we are wondering what is really needed/acceptable wrt throughput and
latency (eg is a
WAL/DB isn't "read intensive". It's more "write intensive" :) use server
SSDs with capacitors to get adequate write performance.
Hi all,
We are thinking about putting our wal/db of hdds/ on ssds. If we would
put the wal of 4 HDDS on 1 SSD as recommended, what type of SSD would
suffice?
We
The tests are measuring differing things, and fio test result of 1.5
MB/s is not bad.
The rados write bench uses by default 4M block size and does 16 threads
and is random in nature, you can change the block size and thread count.
The dd command uses by default 512 block size and and 1
Hi,
your RBD bench and RADOS bench use by default 4MB IO request size while your
FIO is configured for 4KB IO request size.
If you want to compare apple 2 apple (bandwidth) you need to change the FIO IO
request size to 4194304. Plus, you tested a sequential workload with RADOS
bench but
Hello,
I tried to use FIO on RBD device I just created and writing is really
terrible (around 1.5MB/s)
[root@ceph3 tmp]# fio test.fio
rbd_iodepth32: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B,
(T) 4096B-4096B, ioengine=rbd, iodepth=32
fio-3.7
Starting 1 process
Jobs: 1 (f=1):
Hi,
>>dd if=/dev/zero of=/dev/rbd0 writes at 5MB/s -
you are testing with a single thread/iodepth=1 sequentially here.
Then only 1 disk at time, and you have network latency too.
rados bench is doing 16 concurrent write.
Try to test with fio for example, with bigger iodepth, small block/big
Hello,
If this is too long for you, TL;DR; section on the bottom
I created a CEPH cluster made of 3 SuperMicro servers, each with 2 OSD
(WD RED spinning drives) and I would like to optimize the performance of
RBD, which I believe is blocked by some wrong CEPH configuration,
because from my
Hi all,
We are thinking about putting our wal/db of hdds/ on ssds. If we would
put the wal of 4 HDDS on 1 SSD as recommended, what type of SSD would
suffice?
We were thinking of using SATA Read Intensive 6Gbps 1DWPD SSDs.
Does someone has some experience with this configuration? Would we
Maybe this will help you:
https://docs.ceph.com/docs/master/radosgw/multisite/#migrating-a-single-site-system-to-multi-site
___
Clyso GmbH
Am 03.10.2019 um 13:32 schrieb M Ranga Swami Reddy:
Thank you. Do we have a quick document to do this migration?
Thanks
That would accomplish what you are looking for, yes.
Keep in mind that with norebalance that won't stop NEW data from landing there.
It will only keep old data from migrating in. This shouldn't pose too much of
an issue for most use cases.
--
Paul Mezzanini
Sr Systems Administrator /
Hi,our mon is acting up all of a sudden and dying in crash loop with the
following:
2019-10-04 14:00:24.339583 lease_expire=0.00 has v0 lc 4549352
-3> 2019-10-04 14:00:24.335 7f6e5d461700 5
mon.km-fsn-1-dc4-m1-797678@0(leader).paxos(paxos active c 4548623..4549352)
is_readable = 1 -
>
>Try something like the following on each OSD that holds a copy of
>rbd_data.1f114174b0dc51.0974 and see what output you get.
>Note that you can drop the bluestore flag if they are not bluestore
>osds and you will need the osd stopped at the time (set noout). Also
>note,
On Thu, Oct 3, 2019 at 6:46 PM Marc Roos wrote:
>
> >
> >>
> >> I was following the thread where you adviced on this pg repair
> >>
> >> I ran these rados 'list-inconsistent-obj'/'rados
> >> list-inconsistent-snapset' and have output on the snapset. I tried
> to
> >> extrapolate your
Thanks Matt! Really useful configs. I am still on luminous, so I can
forget about this now :( I will try when I am nautilus, I have already
updated my configuration. However it is interesting that in the
configuration nowhere the tenant is specified, so I guess that is being
extracted from
"Path" is either "/" to indicate the top of the tree, or a bucket name
to indicate a limited export for a single bucket. It's not related to
the user at all.
On Thu, Oct 3, 2019 at 10:34 AM Marc Roos wrote:
>
>
> How should a multi tenant RGW config look like, I am not able get this
> working:
Hi Mark,
Here's an example that should work--userx and usery are RGW users
created in different tenants, like so:
radosgw-admin --tenant tnt1 --uid userx --display-name "tnt1-userx" \
--access_key "userxacc" --secret "test123" user create
radosgw-admin --tenant tnt2 --uid usery
We have tried running nfs-ganesha (2.7 - 2.8.1) with FSAL_CEPH backed by
a Nautilus CephFS. Performance when doing metadata operations (ie
anything with small files) is very slow.
On Thu, Oct 3, 2019 at 10:34 AM Marc Roos wrote:
>
>
> How should a multi tenant RGW config look like, I am not able
How should a multi tenant RGW config look like, I am not able get this
working:
EXPORT {
Export_ID=301;
Path = "test:test3";
#Path = "/";
Pseudo = "/rgwtester";
Protocols = 4;
FSAL {
Name = RGW;
User_Id = "test$tester1";
And, just as unexpectedly, things have returned to normal overnight
https://icecube.wisc.edu/~vbrik/graph-1.png
The change seems to have coincided with the beginning of Rados Gateway
activity (before, it was essentially zero). I can see nothing in the
logs that would explain what happened
RGW NFS can support any NFS style of authentication, but users will
have the RGW access of their nfs-ganesha export. You can create
exports with disjoint privileges, and since recent L, N, RGW tenants.
Matt
On Tue, Oct 1, 2019 at 8:31 AM Marc Roos wrote:
>
> I think you can run into problems
So, Ganesha is an NFS gateway, living in userspace. It provides
access via NFS (for any NFS client) to a number of clustered storage
systems, or to local filesystems on it's host. It can run on any
system that has access to the cluster (ceph in this case). One
Ganesha instance can serve quite a
Thank you. Do we have a quick document to do this migration?
Thanks
Swami
On Thu, Oct 3, 2019 at 4:38 PM Paul Emmerich wrote:
> On Thu, Oct 3, 2019 at 12:03 PM M Ranga Swami Reddy
> wrote:
> >
> > Below url says: "Switching from a standalone deployment to a multi-site
> replicated deployment
On Thu, Oct 3, 2019 at 12:03 PM M Ranga Swami Reddy
wrote:
>
> Below url says: "Switching from a standalone deployment to a multi-site
> replicated deployment is not supported.
> https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-rgw-multisite.html
this is wrong,
Dear All,
Many thanks to Brad and Mattia for good advice.
I was away for two days, in the meantime the pg has fixed itself.
I'm not complaining, but it's strange...
Looking at the OSD logs, we see the previous repair fail. Then a routine
scrub appears to fix the issue. The same thing happened
Below url says: "Switching from a standalone deployment to a multi-site
replicated deployment is not supported.
https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-rgw-multisite.html
Please advise.
On Thu, Oct 3, 2019 at 3:28 PM M Ranga Swami Reddy
wrote:
> Hi,
>
Hi,
Iam using the 2 ceph clusters in diff DCs (away by 500 KM) with ceph
12.2.11 version.
Now, I want to setup rgw multisite using the above 2 ceph clusters.
is it possible? if yes, please share good document to do the same.
Thanks
Swami
___
ceph-users
Thank you Robin.
Looking at the video it doesn't seem like a fix is anywhere near ready.
Am I correct in concluding that Ceph is not the right tool for my use-case?
Cheers,
Christian
On Oct 3 2019, at 6:07 am, Robin H. Johnson wrote:
> On Wed, Oct 02, 2019 at 01:48:40PM +0200, Christian
901 - 1000 of 52714 matches
Mail list logo
