;fix themselves" without
apparent cause!
Graham
On 4/29/19 12:12 PM, Graham Allan wrote:
I think I need a second set of eyes to understand some unexpected data
movement when adding new OSDs to a cluster (Luminous 12.2.11).
Our cluster ran low on space sooner than expected; so as a stopg
I think I need a second set of eyes to understand some unexpected data
movement when adding new OSDs to a cluster (Luminous 12.2.11).
Our cluster ran low on space sooner than expected; so as a stopgap I
recommissioned a couple of older storage nodes while we get new hardware
purchases under
as a
mirror; wait for it to sync, then break the mirror and remove the
original disk.
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users
the 12.2.9 nodes kept to wait for a future release - or wait on all?
Thanks, Graham
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
are expected to work.
Currently using Luminous 12.2.8, if it matters.
Graham
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
s-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users mailing list
produce nice output? If so, it may be playing a role in the
delay as well.
Eric
On 9/26/18 5:27 PM, Graham Allan wrote:
I have one user bucket, where inexplicably (to me), the bucket takes an
eternity to list, though only on the top level. There are two
subfolders, each of which lists individual
01:44 PM, Gregory Farnum wrote:
On Thu, Oct 11, 2018 at 3:22 PM Graham Allan
As the osd crash implies, setting "nobackfill" appears to let all the
osds keep running and the pg stays active and can apparently serve data.
If I track down the object referenced below in
but one I have seen implicated in similar
crash logs for otehr OSDs and the etag again does not match; the other I
have not seen in crash logs and does generate a matching etag.
Opened a tracker issue for this: http://tracker.ceph.com/issues/36411
Graham
On 10/09/2018 06:55 PM, Graham Allan wrote:
On
On 10/09/2018 01:14 PM, Graham Allan wrote:
On 10/9/2018 12:19 PM, Gregory Farnum wrote:
I think unfortunately the easiest thing for you to fix this will be to
set the min_size back to 4 until the PG is recovered (or at least has
5 shards done). This will be fixed in a later version
On 10/9/2018 12:19 PM, Gregory Farnum wrote:
On Wed, Oct 3, 2018 at 10:18 AM Graham Allan <mailto:g...@umn.edu>> wrote:
However I have one pg which is stuck in state remapped+incomplete
because it has only 4 out of 6 osds running, and I have been unable to
bring the mi
Oops, by "periods" I do of course mean "intervals"...!
On 10/8/2018 4:57 PM, Graham Allan wrote:
I'm still trying to find a way to reactivate this one pg which is
incomplete. There are a lot of periods in its history based on a
combination of a peering storm a
/e72b4461fb315983ae9a62cbbcd851d4/raw/0d30ceb315dd5567cb05fd0dc3e2e2c4975d8c01/pg70.b1c-query.txt
(Out of curiosity, is there any way to relate the first and last numbers
in an interval to an actual timestamp?)
Thanks,
Graham
On 10/03/2018 12:18 PM, Graham Allan wrote:
Following on from my
rying but there isn't a lot of documentation on the
operations available in ceph-objectstore-tool.
I also know of the option "osd_find_best_info_ignore_history_les" but
little of what it actually does, other than being "dangerous". There are
many past intervals listed
ome "radosgw-admin bucket check [--fix]" commands
with no change.
Graham
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
On 09/17/2018 04:33 PM, Gregory Farnum wrote:
On Mon, Sep 17, 2018 at 8:21 AM Graham Allan <mailto:g...@umn.edu>> wrote:
Looking back through history it seems that I *did* override the
min_size
for this pool, however I didn't reduce it - it used to have min_size 2!
On 09/14/2018 02:38 PM, Gregory Farnum wrote:
On Thu, Sep 13, 2018 at 3:05 PM, Graham Allan wrote:
However I do see transfer errors fetching some files out of radosgw - the
transfer just hangs then aborts. I'd guess this probably due to one pg stuck
down, due to a lost (failed HDD) osd. I
sgw
disabled).
So if 98 were marked lost would it roll back to the prior interval? I am
not certain how to interpret this information!
Running luminous 12.2.7 if it makes a difference.
Thanks as always for pointers,
Graham
On 07/18/2018 02:35 PM, Graham Allan wrote:
Like many, we have a
ake" command, so this shouldn't be significant.
At least I hope not. The relationship between the crush rule and the ec
profile, when changing the crush rule on an existing ec pool, is still
not very clear to me...
Graham
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
__
g time ago under hammer?).
Thanks for any feedback,
Graham
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
a3355f364f3
Author: David Zafman <dzaf...@redhat.com>
Date: Mon Oct 9 08:19:21 2017 -0700
osd, mon: Add new pg states recovery_unfound and backfill_unfound
Signed-off-by: David Zafman <dzaf...@redhat.com>
On 2/16/18 1:40 PM, Gregory Farnum wrote:
On Fri, Feb 16, 2018 a
fo is either failing or incorrect!
Did you run a local build w/ the linked patch? I think that would have
more effect than
I did just build a local copy of 12.2.2 with the patch - and it does
seem to fix it.
Thanks!
Graham
--
Graham Allan
Minnesota Supercomputing Inst
AAwIXBgAAAGJseW5jaAkAAABCZW4gTHluY2gDA2IBAQYAAABibHluY2gPAQYAAABibHluY2gEAzcCAgQABgAAAGJseW5jagIEDwkAAABCZW4gTHluY2gAAA=="
},
{
"key": "user.rgw.idtag",
"
ed around
10/2016 - I suspect the placement policies were incorrect for a short
time due to confusion over the hammer->jewel upgrade (the
realm/period/zonegroup/zone conversion didn't really go smoothly!)
On 02/16/2018 11:39 PM, Robin H. Johnson wrote:
On Fri, Feb 16, 2018 at 07:06:21PM
they are not in this state. It is a 4+2 EC pool, so I
would think it possible to reconstruct any missing EC chunks.
It's an extensive problem; while I have been focusing on examining a
couple of specific pgs, the pool in general is showing 2410 pgs
inconsistent (out of 4096).
Graham
--
Graham
;: ".rgw.control",
"gc_pool": ".rgw.gc",
"lc_pool": ".log:lc",
"log_pool": ".log",
"intent_log_pool": ".intent-log",
"usage_log_pool": ".usage",
"reshard_pool": ".log:reshard",
"user_keys_pool": ".users",
"user_email_pool": ".users.email",
"user_swift_pool": ".users.swift",
"user_uid_pool": ".users.uid",
"system_key": {
"access_key": "",
"secret_key": ""
},
"placement_pools": [
{
"key": "default-placement",
"val": {
"index_pool": ".rgw.buckets.index",
"data_pool": ".rgw.buckets",
"data_extra_pool": ".rgw.buckets.extra",
"index_type": 0,
"compression": ""
}
},
{
"key": "ec42-placement",
"val": {
"index_pool": ".rgw.buckets.index",
"data_pool": ".rgw.buckets.ec42",
"data_extra_pool": ".rgw.buckets.extra",
"index_type": 0,
"compression": ""
}
}
],
"metadata_heap": ".rgw.meta",
"tier_config": [],
"realm_id": "dbfd45d9-e250-41b0-be3e-ab9430215d5b"
}
Graham
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
On 02/16/2018 12:31 PM, Graham Allan wrote:
If I set debug rgw=1 and demug ms=1 before running the "object stat"
command, it seems to stall in a loop of trying communicate with osds for
pool 96, which is .rgw.control
10.32.16.93:0/2689814946 --> 10.31.0.68:6818/8969 --
osd_o
On 02/15/2018 05:33 PM, Gregory Farnum wrote:
On Thu, Feb 15, 2018 at 3:10 PM Graham Allan <g...@umn.edu
<mailto:g...@umn.edu>> wrote:
A lot more in xattrs which I won't paste, though the keys are:
> root@cephmon1:~# ssh ceph03 find
/var/lib/ceph/osd/c
overy_wait+inconsistent by "ceph health detail" is shown as
active+recovering+inconsistent by "ceph pg list". That makes more sense
to me - "recovery_wait" implied to me that it was waiting for recovery
to start, while "recovering" explains why the problem migh
Hi Greg,
On 02/14/2018 11:49 AM, Gregory Farnum wrote:
On Tue, Feb 13, 2018 at 8:41 AM Graham Allan <g...@umn.edu
<mailto:g...@umn.edu>> wrote:
I'm replying to myself here, but it's probably worth mentioning that
after this started, I did bring back the failed
problem being found.
Graham
On 02/12/2018 06:26 PM, Graham Allan wrote:
Hi,
For the past few weeks I've been seeing a large number of pgs on our
main erasure coded pool being flagged inconsistent, followed by them
becoming active+recovery_wait+inconsistent with unfound objects. The
cluster
at thread, I'm not finding any other files with
duplicate names in the hierarchy.
I'm not sure there's much else I can do besides record the names of any
unfound objects before resorting to "mark_unfound_lost delete" - any
suggestions for further research?
Thanks,
Graham
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
h-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
? Or is object
lifecycle functionality available as soon as radosgw is upgraded?
Thank you
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Graham Allan
Minnesota Supercomputing Institute - g
that make sense?
Thanks,
Graham
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
On 12/14/2017 04:00 AM, Martin Emrich wrote:
Hi!
Am 13.12.17 um 20:50 schrieb Graham Allan:
After our Jewel to Luminous 12.2.2 upgrade, I ran into some of the
same issues reported earlier on the list under "rgw resharding
operation seemingly won't end".
Yes, that were/are my
ts (multi-million objects in a single shard).
Thanks for any pointers,
Graham
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
from /etc/logrotate.d/ceph) from being SIGHUPed,
and to rotate the logs manually from time to time and completely restarting the radosgw
processes one after the other on my radosgw cluster.
Regards,
Martin
Am 08.12.17, 18:58 schrieb "ceph-users im Auftrag von Graham Allan&quo
restarting, if it's useful to open an issue.
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
crush remove osd.2
device 'osd.2' does not appear in the crush map
so I wonder where it's getting this warning from, and if it's erroneous,
how can I clear it?
Graham
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users
On 12/06/2017 03:20 AM, Wido den Hollander wrote:
Op 5 december 2017 om 18:39 schreef Richard Hesketh
<richard.hesk...@rd.bbc.co.uk>:
On 05/12/17 17:10, Graham Allan wrote:
On 12/05/2017 07:20 AM, Wido den Hollander wrote:
I haven't tried this before but I expect it to work, but I
one machine at a time to
reinstall with CentOS 7 and Bluestore. I too don't see any reason the
mixed Jewel/Luminous cluster wouldn't work, but still felt less
comfortable with extending the upgrade duration.
Graham
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
s.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
s.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
read bucket contents.
Admittedly I have no experience with AWS bucket policies so I could be
doing something dumb...
Thanks,
Graham
On 07/17/2017 06:33 PM, Graham Allan wrote:
Thanks for the update. I saw there was a set of new 12.1.1 packages
today so I updated to these (appears to cont
ce": ["arn:aws:s3:::gta/*"]
}
]
}
but...
% s3cmd setpolicy s3policy s3://gta
ERROR: S3 error: 400 (InvalidArgument)
I have "debug rgw = 20" but nothing revealing in the logs.
Do you see anything obviously wrong in my policy file?
Thanks,
Graham
On 07/12/2017 11:2
the same crash
- { "AWS": "arn:aws:iam::lemming:gta4"} causes principal discarded
of course... that is the user I am trying to grant access to. Possibly
the problem might be the blank tenant for the bucket owner?
Thanks,
Graham
On 07/12/2017 02:53 PM, Adam C. Emerson wrote
radosgw seems to accept without
discarding, however user gta2 still has no access.
So tried setting the principal to {"AWS": "arn:aws:iam:::user/gta2"}
This just resulted in a crash dump from radosgw... in summary
rgw_iam_policy.cc: In function 'boost::optional
rgw::IAM::par
0602c7-224d-4a12-884a-58e5dc9a87e9
So it seems like some kind of race condition. It does happen
consistently at reboot for all these systems, though the particular OSDs
affected change each time.
Any ideas how to troubleshoot further... or should I open a bug report?
Thanks!
Graham
--
Grah
/ceph-users-ceph.com
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
ed.
So, I think I understand why it may not have worked before, but the
goalposts seem to have changed to a new problem.
Would appreciate any ideas...
Graham
On 06/08/2017 11:54 AM, Graham Allan wrote:
Sorry I didn't get to reply until now. The thing is I believe I *do*
have a lifecycle configu
obably need to add a
lifecycle configuration using the S3 API. It's not automatic and has to
be added per-bucket.
Here's some sample code for doing so: http://tracker.ceph.com/issues/19587
-Ben
On Tue, Jun 6, 2017 at 9:07 AM, Graham Allan <g...@umn.edu
<mailto:g...@umn.edu>> wrote:
I
/tracker.ceph.com/issues/20177%5D>
As it's my first one, hope it's ok as it is...
Thanks & regards
Anton
Gesendet: Samstag, 03. Juni 2017 um 00:14 Uhr
Von: "Yehuda Sadeh-Weinraub" <yeh...@redhat.com
<mailto:yeh...@redhat.com&g
one
2017-05-19 15:28:46.453431 7feddc240c80 2 all 8 watchers are set, enabling
cache
2017-05-19 15:28:46.614991 7feddc240c80 2 removed watcher, disabling cache
"radosgw-admin lc list" seems to return "empty" output:
# radosgw-admin lc list
[]
Is t
On 03/27/2017 03:13 PM, Graham Allan wrote:
I'm following up to myself here, but I'd love to hear if anyone knows
how the global quotas can be set in jewel's radosgw. I haven't found
anything which has an effect - the documentation says to use:
radosgw-admin region-map get > regionmap.json
...e
map set < regionmap.json
but this has no effect on jewel. There doesn't seem to be any analogous
function in the "period"-related commands which I think would be the
right place to look for jewel.
Am I missing something, or should I open a bug?
Graham
On 03/21/2017 03:18 PM, Gr
nconsistency.
Thanks for any suggestions,
G.
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
On 03/17/2017 11:47 AM, Casey Bodley wrote:
On 03/16/2017 03:47 PM, Graham Allan wrote:
This might be a dumb question, but I'm not at all sure what the
"global quotas" in the radosgw region map actually do.
It is like a default quota which is applied to all users or buckets,
with
all users and buckets in the region/zonegroup?
Graham
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
On Thu, Dec 8, 2016 at 5:19 AM, Francois Lafont <
francois.lafont.1...@gmail.com> wrote:
> On 12/08/2016 11:24 AM, Ruben Kerkhof wrote:
>
> > I've been running this on one of my servers now for half an hour, and
> > it fixes the issue.
>
> It's the same for me. ;)
>
> ~$ ceph -v
> ceph version
~10nUehK5BnyXdhhiOqTL2JdpLfDCd0k.11_76
lemming
edit "num_shards": 32
# radosgw-admin metadata put bucket.instance:tcga:default.712449.19 < tcga.json
and this bucket is now visible again! Thanks so much!
I wonder how this happened. It looks like this affects ~25/680 buckets.
Graham
--
On 11/21/2016 04:44 PM, Yehuda Sadeh-Weinraub wrote:
On Mon, Nov 21, 2016 at 2:42 PM, Graham Allan <g...@umn.edu> wrote:
Following up to this (same problem, looking at it with Jeff)...
There was definite confusion with the zone/zonegroup/realm/period changes
during the hammer->jewe
with a couple of the rados gateway servers and
inadvertently deployed older Hammer versions of the radosgw instances.
This configuration was running for a couple of days. We removed the Hammer
versions and re-deployed the Jewel versions of the radosgw.S3cmd
querying of some of the buckets
://tracker.ceph.com/issues/17371
If an hammer radosgw-admin runs on the jewel radosgw it corrupts the
configuration.
We are working on a fix for that.
Orit
On Fri, Oct 7, 2016 at 9:37 PM, Graham Allan <g...@umn.edu> wrote:
Dear Orit,
On 10/07/2016 04:21 AM, Orit Wasserman wrote:
Hi,
On Wed, Oct 5, 2016
# radosgw-admin zone get --rgw-zone=default
{
"id": "default",
"name": "default",
"domain_root": ".rgw",
"control_pool": ".rgw.control",
"gc_pool": ".rgw.gc",
"log_pool": ".log",
"intent_log_pool": ".intent-log",
"usage_log_pool": ".usage",
"user_keys_pool": ".users",
"user_email_pool": ".users.email",
"user_swift_pool": ".users.swift",
"user_uid_pool": ".users.uid",
"system_key": {
"access_key": "",
"secret_key": ""
},
"placement_pools": [],
"metadata_heap": ".rgw.meta",
"realm_id": ""
}
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
and
user clients can connect.
Hope that helps
andrei
- Original Message -
From: "Graham Allan" <g...@umn.edu>
To: "ceph-users" <ceph-users@lists.ceph.com>
Sent: Thursday, 6 October, 2016 20:04:38
Subject: Re: [ceph-users] unable to start radosgw afte
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http
Does anyone know if this issue was corrected in Hammer 0.94.8?
http://tracker.ceph.com/issues/15002
It's marked as resolved but I don't see it listed in the release notes.
G.
--
Graham Allan
Systems Researcher - Minnesota Supercomputing Institute - g...@umn.edu
69 matches
Mail list logo
