from:"Andrew Walker"

A reboot of the host has fixed the problem but I still want to find the root 
cause. 

Looking at the logs I can see the original mon went down because the docker 
engine shutdown in response to a network event. That network event seems to 
appears to be systemd wait-on-network timeout related and an daily apt updates 
check happening at the same time. 

When the mon was rebooted, and came back up, this seemed to trigger the OSDs on 
a separate server to shutdown.  Only OSD containers shutdown, not mon/mgr or 
mds containers. Again, this seems to be a requested shutdown rather than a 
crash….

Need to some more digging…..

Any thoughts would be appreciated. 

A

Sent from my iPhone

On 6 Aug 2021, at 09:20, David Caro  wrote:

On 08/06 07:59, Andrew Walker-Brown wrote:
> Hi Marc,
> 
> Yes i’m probably doing just that.
> 
> The ceph admin guides aren’t exactly helpful on this.  The cluster was 
> deployed using cephadm and it’s been running perfectly until now.
> 
> Wouldn’t running “journalctl -u ceph-osd@5” on host ceph-004 show me the logs 
> for osd.5 on that host?

On my containerized setup, the services that cephadm created are:

dcaro@node1:~ $ sudo systemctl list-units | grep ceph
 ceph-d49b287a-b680-11eb-95d4-e45f010c03a8@crash.node1.service  
   loaded active 
running   Ceph crash.node1 for d49b287a-b680-11eb-95d4-e45f010c03a8
 ceph-d49b287a-b680-11eb-95d4-e45f010c03a8@mgr.node1.mhqltg.service 
   loaded active 
running   Ceph mgr.node1.mhqltg for d49b287a-b680-11eb-95d4-e45f010c03a8
 ceph-d49b287a-b680-11eb-95d4-e45f010c03a8@mon.node1.service
   loaded active 
running   Ceph mon.node1 for d49b287a-b680-11eb-95d4-e45f010c03a8
 ceph-d49b287a-b680-11eb-95d4-e45f010c03a8@osd.3.service
   loaded active 
running   Ceph osd.3 for d49b287a-b680-11eb-95d4-e45f010c03a8
 ceph-d49b287a-b680-11eb-95d4-e45f010c03a8@osd.7.service
   loaded active 
running   Ceph osd.7 for d49b287a-b680-11eb-95d4-e45f010c03a8
 system-ceph\x2dd49b287a\x2db680\x2d11eb\x2d95d4\x2de45f010c03a8.slice  
   loaded active 
activesystem-ceph\x2dd49b287a\x2db680\x2d11eb\x2d95d4\x2de45f010c03a8.slice
 ceph-d49b287a-b680-11eb-95d4-e45f010c03a8.target   
   loaded active 
activeCeph cluster d49b287a-b680-11eb-95d4-e45f010c03a8
 ceph.target
   loaded active 
activeAll Ceph clusters and services

where the string after 'ceph-' is the fsid of the cluster.
Hope that helps (you can use the systemctl list-units also to search the 
specific ones on yours).

> 
> Cheers,
> A
> 
> 
> 
> 
> 
> Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
> 
> From: Marc<mailto:m...@f1-outsourcing.eu>
> Sent: 06 August 2021 08:54
> To: Andrew Walker-Brown<mailto:andrew_jbr...@hotmail.com>; 
> ceph-users@ceph.io<mailto:ceph-users@ceph.io>
> Subject: RE: All OSDs on one host down
> 
>> 
>> I’ve tried restarting on of the osds but that fails, journalctl shows
>> osd not found.not convinced I’ve got the systemctl command right.
>> 
> 
> You are not mixing 'not container commands' with 'container commands'. As in, 
> if you execute this journalctl outside of the container it will not find 
> anything of course.
> 
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

-- 
David Caro
SRE - Cloud Services
Wikimedia Foundation <https://wikimediafoundation.org/>
PGP Signature: 7180 83A2 AC8B 314F B4CE  1171 4071 C7E1 D262 69C3

"Imagine a world in which every single human being can freely share in the
sum of all knowledge. That's our commitment."
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: All OSDs on one host down

This is what I get from cephadm logs for the specific OSDin fact all OSDs 
on that host.

bash[2681]: Stalls(count): 0 level0_slowdown, 0 
level0_slowdown_with_compaction, 0 level0_numfiles, 0 
level0_numfiles_with_compaction, 0 stop for pending_compa>
Aug 06 06:52:37 uk1-ceph-004 bash[2681]: ** File Read Latency Histogram By 
Level [default] **
Aug 06 06:53:12 uk1-ceph-004 bash[2681]: debug 2021-08-06T06:53:12.035+ 
7fdf46f0a700 -1 received  signal: Terminated from Kernel ( Could be generated 
by pthread_kill(), raise(), abort(>
Aug 06 06:53:12 uk1-ceph-004 bash[2681]: debug 2021-08-06T06:53:12.035+ 
7fdf46f0a700 -1 osd.14 12892 *** Got signal Terminated ***
Aug 06 06:53:12 uk1-ceph-004 bash[2681]: debug 2021-08-06T06:53:12.035+ 
7fdf46f0a700 -1 osd.14 12892 *** Immediate shutdown (osd_fast_shutdown=true) ***
Aug 06 06:53:19 uk1-ceph-004 docker[1156357]: Error response from daemon: No 
such container: ceph-ce531468-7dc9-11eb-b20e-0dcf90f47aea-osd.14
Aug 06 06:53:19 uk1-ceph-004 bash[1156607]: docker: error during connect: Post 
http://%2Fvar%2Frun%2Fdocker.sock/v1.41/containers/create?name=ceph-ce531468-7dc9-11eb-b20e-0dcf90f47aea-osd.>

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

From: David Caro<mailto:dc...@wikimedia.org>
Sent: 06 August 2021 09:20
To: Andrew Walker-Brown<mailto:andrew_jbr...@hotmail.com>
Cc: Marc<mailto:m...@f1-outsourcing.eu>; 
ceph-users@ceph.io<mailto:ceph-users@ceph.io>
Subject: Re: [ceph-users] Re: All OSDs on one host down

On 08/06 07:59, Andrew Walker-Brown wrote:
> Hi Marc,
>
> Yes i’m probably doing just that.
>
> The ceph admin guides aren’t exactly helpful on this.  The cluster was 
> deployed using cephadm and it’s been running perfectly until now.
>
> Wouldn’t running “journalctl -u ceph-osd@5” on host ceph-004 show me the logs 
> for osd.5 on that host?

On my containerized setup, the services that cephadm created are:

dcaro@node1:~ $ sudo systemctl list-units | grep ceph
  ceph-d49b287a-b680-11eb-95d4-e45f010c03a8@crash.node1.service 
loaded active 
running   Ceph crash.node1 for d49b287a-b680-11eb-95d4-e45f010c03a8
  ceph-d49b287a-b680-11eb-95d4-e45f010c03a8@mgr.node1.mhqltg.service
loaded active 
running   Ceph mgr.node1.mhqltg for d49b287a-b680-11eb-95d4-e45f010c03a8
  ceph-d49b287a-b680-11eb-95d4-e45f010c03a8@mon.node1.service   
loaded active 
running   Ceph mon.node1 for d49b287a-b680-11eb-95d4-e45f010c03a8
  ceph-d49b287a-b680-11eb-95d4-e45f010c03a8@osd.3.service   
loaded active 
running   Ceph osd.3 for d49b287a-b680-11eb-95d4-e45f010c03a8
  ceph-d49b287a-b680-11eb-95d4-e45f010c03a8@osd.7.service   
loaded active 
running   Ceph osd.7 for d49b287a-b680-11eb-95d4-e45f010c03a8
  system-ceph\x2dd49b287a\x2db680\x2d11eb\x2d95d4\x2de45f010c03a8.slice 
loaded active 
activesystem-ceph\x2dd49b287a\x2db680\x2d11eb\x2d95d4\x2de45f010c03a8.slice
  ceph-d49b287a-b680-11eb-95d4-e45f010c03a8.target  
loaded active 
activeCeph cluster d49b287a-b680-11eb-95d4-e45f010c03a8
  ceph.target   
loaded active 
activeAll Ceph clusters and services

where the string after 'ceph-' is the fsid of the cluster.
Hope that helps (you can use the systemctl list-units also to search the 
specific ones on yours).


>
> Cheers,
> A
>
>
>
>
>
> Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
>
> From: Marc<mailto:m...@f1-outsourcing.eu>
> Sent: 06 August 2021 08:54
> To: Andrew Walker-Brown<mailto:andrew_jbr...@hotmail.com>; 
> ceph-users@ceph.io<mailto:ceph-users@ceph.io>
> Subject: RE: All OSDs on one host down
>
> >
> > I’ve tried restarting on of the osds but that fails, journalctl shows
> > osd not found.not convinced I’ve got the systemctl command right.
> >
>
> You are not mixing 'not container commands' with 'container commands'. As in, 
> if you execute this journalctl outside of the container it will not find 
> anything of course.
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

--
David Caro
SRE - Clo

[ceph-users] Re: All OSDs on one host down

Hi Marc,

Yes i’m probably doing just that.

The ceph admin guides aren’t exactly helpful on this.  The cluster was deployed 
using cephadm and it’s been running perfectly until now.

Wouldn’t running “journalctl -u ceph-osd@5” on host ceph-004 show me the logs 
for osd.5 on that host?

Cheers,
A





Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

From: Marc<mailto:m...@f1-outsourcing.eu>
Sent: 06 August 2021 08:54
To: Andrew Walker-Brown<mailto:andrew_jbr...@hotmail.com>; 
ceph-users@ceph.io<mailto:ceph-users@ceph.io>
Subject: RE: All OSDs on one host down

>
> I’ve tried restarting on of the osds but that fails, journalctl shows
> osd not found.not convinced I’ve got the systemctl command right.
>

You are not mixing 'not container commands' with 'container commands'. As in, 
if you execute this journalctl outside of the container it will not find 
anything of course.


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] All OSDs on one host down

Hi all,

Bit of a panic. Woke this morning to find on of my dedicated mon hosts showing 
as down.  I did a reboot on the host and it came back up fine (ceph-005).

Then all the OSDs (5) on host ceph-004 went down.  This host is also a mon and 
the mon daemon is showing as up.

I’m running octopus using containers.

I’ve tried restarting on of the osds but that fails, journalctl shows osd not 
found.not convinced I’ve got the systemctl command right.

Any ideas where to start please.

Regards,

Andrew

Sent from Mail for Windows 10

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Nic bonding (lacp) settings for ceph

2021-06-28 Thread Andrew Walker-Brown

HI,

I think ad_select is only relevant in the scenario below I.e where you have 
more than one port-channel being presented to the Linux bond.  So below, you 
have 2 port channels, one from each switch, but at the Linux side all the ports 
involved are slaves in the same bond.  In your scenario it sounds like you just 
have one switch with one port-channel to one bond on Linux.  So in the case of 
ad_select, I doubt it has any impact.  The main thing will be the 
xmit-hash-policy on both the switches and Linux.  FWIW, I use layer3+4 on Linux 
and something very close to that on my S series switches, and both 10G links 
get used pretty well.  (below was lifted from a stackexchange thread)

.---.   .---.

|  Switch1  |   |  Switch2  |

'-=---=-'   '-=---=-'

  |   |   |   |

  |   |   |   |

.-=.--=---.---=--.=-.

| eth0 | eth1 | eth2 | eth3 |

|---|

|   bond0   |

'---'

Where each switch has its two ports configured in a PortChannel, the Linux end 
with the LACP bond will negotiate two Aggregator IDs:

Aggregator ID 1

 - eth0 and eth1

Aggregator ID 2

 - eth2 and eth3

And the switches will have a view completely separate of each other.

Switch 1 will think:

Switch 1

 PortChannel 1

 - port X

 - port Y

Switch 2 will think:

Switch 2

 PortChannel 1

 - port X

 - port Y

>From the Linux system with the bond, only one Aggregator will be used at a 
>given time, and will fail over depending on ad_select.

So assuming Aggregator ID 1 is in use, and you pull eth0's cable out, the 
default behaviour is to stay on Aggregator ID 1.

However, Aggregator ID 1 only has 1 cable, and there's a spare Aggregator ID 2 
with 2 cables - twice the bandwidth!

If you use ad_select=count or ad_select=bandwidth, the active Aggregator ID 
fails over to an Aggregator with the most cables or the most bandwidth.

Note that LACP mandates an Aggregator's ports must all be the same speed and 
duplex, so I believe you could configure one Aggregator with 1Gbps ports and 
one Aggregator with 10Gbps ports, and have intelligent selection depending on 
whether you have 20/10/2/1Gbps available.

Sent from Mail for Windows 10

From: mhnx
Sent: 28 June 2021 18:46
To: Marc 'risson' Schmitt
Cc: Ceph Users
Subject: [ceph-users] Re: Nic bonding (lacp) settings for ceph

Thanks for the answer.
I'm into ad_select bandwitdh because we use osd nodes as rgw gateways, VMs
and different applications.

I have seperate cluster (10+10Gbe) and public (10+10Gbe) network.
I tested stable, bandwitdh and count. Results are clearly good with
bandwitdh. Count is the worst option.
But I wonder if bandwitdh calculation has any effect on the network delay?
If it is then I will return to stable. I don't know now but when i think
about it if every time bonding driver needs to calculate bandwitdh and
decide it should add some cpu power and delay. If it has no effect then
bandwitdh will improve distribution better.

Now I know that I have to use 3+4 but still couldn't decide on ad_select.
Bandwitdh or stable?
Can we discuss it please?

28 Haz 2021 Pzt 20:15 tarihinde Marc 'risson' Schmitt 
şunu yazdı:

> Hi,
>
> On Sat, 26 Jun 2021 16:47:19 +0300
> mhnx  wrote:
> > I've changed ad_select to bandwitdh and both nic is in use now but
> > layer2 hash prevents dual nic usage for between two nodes (because
> > layer2 using only Mac ).
>
> As I understand it, setting ad_select to bandwidth is only going to be
> useful if you have several link aggregates in the same bond, like when
> you are connected in LACP to multiple (non-stacked) switches.
>
> > People advice using layer2+3 for best performance but it has no
> > effect on osds because mac and ip is the same.
> > I've tried layer3+4 to split by ports instead mac and it works. But i
> > dont know what will the effect and also my switch is layer2.
>
> We are setting layer3+4 on both our servers and our switches.
>
> Regards,
>
> --
> Marc 'risson' Schmitt
> CRI - EPITA
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Managers dieing?

2021-06-17 Thread Andrew Walker-Brown

Changing pg_num and pgp_num manually can be a useful tool.  Just remember that 
they need to be factor of 2, don’t increase or decease more than a couple of 
steps e.g. 64 to 128 or 256….but not to 1024 etc. 

I had a situation where a couple of OSDs got quite full. I added more capacity 
but the rebalance got stuck as there wasn’t enough space on one of the full’ish 
OSDs to put a pg. 

I increased the pg_num and pgp_num (doubled)…this effectively made the pg’s 
smaller so Ceph could finish the rebalance, it could squeeze a pg in the space 
left on the OSDs. Once that was done I just turned autoscale back on. 

Sent from my iPhone

On 17 Jun 2021, at 18:13, Peter Childs  wrote:

Found the issue in the end  I'd managed to kill the autoscaling features by
playing with pgp_num and pg_num and it was getting confusing. I fixed it in
the end by reducing pg_num on some of my test pools and the manager woke up
and started working again.

It was not clear as to what I'd done to kill it but once I'd figured out
what was crashing it was possible to figure out what was going to help it.

So I've just learnt, Don't play with pgp_num and pg_num and let the
autoscaling feature just work. setting the target size or ratio is probably
better.

I like ceph its very different to Spectrum Scale which I've used for years,
but for now its different tools to resolve different issues.

Must get around to doing something with what I've learnt so far.

Peter

On Thu, 17 Jun 2021 at 17:53, Eugen Block  wrote:

> Hi,
> 
> don't give up on Ceph. ;-)
> 
> Did you try any of the steps from the troubleshooting section [1] to
> gather some events and logs? Could you share them, and maybe also some
> more details about that cluster? Did you enable any non-default mgr
> modules? There have been a couple reports related to mgr modules.
> 
> Regards
> Eugen
> 
> [1] 
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2Fcephadm%2Ftroubleshooting%2Fdata=04%7C01%7C%7C1b5c18d7d4324fa1f9d708d931b34476%7C84df9e7fe9f640afb435%7C1%7C0%7C637595468280730494%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=MwFUwPG4hX3IJ4DvRdPMie25KmJ9XzaAgzMw3v%2FnRwg%3Dreserved=0
> 
> 
> Zitat von Peter Childs :
> 
>> Lets try to stop this message turning into a mass moaning session about
>> Ceph and try and get this newbie able to use it.
>> 
>> I've got a Ceph Octopus cluster, its relatively new and deployed using
>> cephadm.
>> 
>> It was working fine, but now the managers start up run for about 30
> seconds
>> and then die, until systemctl gives up and I have to reset-fail them to
> get
>> them to try again, when they fail.
>> 
>> How do I work out why and get them working again?
>> 
>> I've got 21 nodes and was looking to take it up to 32 over the next few
>> weeks, but that is going to be difficult if the managers are not working.
>> 
>> I did try Pacific and I'm happy to upgrade but that failed to deploy more
>> than 6 osd's and I gave up and went back to Octopus.
>> 
>> I'm about to give up on Ceph because it looks like its really really
>> "fragile" and debugging what's going wrong is really difficult.
>> 
>> I guess I could give up on cephadm and go with a different provisioning
>> method but I'm not sure where to start on that.
>> 
>> Thanks in advance.
>> 
>> Peter.
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
> 
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
> 
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Fwd: Re: Issues with Ceph network redundancy using L2 MC-LAG

2021-06-16 Thread Andrew Walker-Brown

Ideally you’d want to have it the transmit hash the same...but so long load is 
being pretty evenly spread over all the links in the lag, then you’re fine.

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

From: mabi<mailto:m...@protonmail.ch>
Sent: 16 June 2021 09:29
To: Andrew Walker-Brown<mailto:andrew_jbr...@hotmail.com>
Cc: huxia...@horebdata.cn<mailto:huxia...@horebdata.cn>; Joe 
Comeau<mailto:joe.com...@hli.ubc.ca>; ceph-users<mailto:ceph-users@ceph.io>
Subject: Re: [ceph-users] Re: Fwd: Re: Issues with Ceph network redundancy 
using L2 MC-LAG

‐‐‐ Original Message ‐‐‐
On Wednesday, June 16, 2021 10:18 AM, Andrew Walker-Brown 
 wrote:

> With active mode, you then have a transmit hashing policy, usually set 
> globally.
>
> On Linux the bond would be set as ‘bond-mode 802.3ad’ and then 
> ‘bond-xmit-hash-policy layer3+4’ - or whatever hashing policy you want.

Does the transmit hash policy need to be the same on the Linux server and on 
the switch side? or can they be different?

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Fwd: Re: Issues with Ceph network redundancy using L2 MC-LAG

2021-06-16 Thread Andrew Walker-Brown

Depends on when you configure the switch port. For dell :

Interface Ethernet 1/1/20
No switchport
Channel-group 10 mode active
!

‘Mode active’ set it as a dynamic lacp lag. Otherwise it would be ‘mode static’

With active mode, you then have a transmit hashing policy, usually set 
globally. 

On Linux the bond would be set as ‘bond-mode 802.3ad’ and then 
‘bond-xmit-hash-policy layer3+4’ - or whatever hashing policy you want. 

Sent from my iPhone

On 16 Jun 2021, at 08:57, huxia...@horebdata.cn wrote:

Is it true that MC-LAG and 803.2ad, by its default, are working on 
active-active. 

What else should i take care to ensure fault tolerance when one path is bad?

best regards,

samuel

huxia...@horebdata.cn

From: Joe Comeau
Date: 2021-06-15 23:44
To: ceph-users@ceph.io
Subject: [ceph-users] Fwd: Re: Issues with Ceph network redundancy using L2 
MC-LAG
We also run with Dell VLT switches (40 GB)
everything is active/active, so multiple paths as Andrew describes in
his config
Our config allows us:
  bring down one of the switches for upgrades
  bring down an iscsi gatway for patching
all the while at least one path is up and servicing
Thanks Joe

>>> Andrew Walker-Brown  6/15/2021 10:26 AM
>>> 
With an unstable link/port you could see the issues you describe.  Ping
doesn’t have the packet rate for you to necessarily have a packet in
transit at exactly the same time as the port fails temporarily.  Iperf
on the other hand could certainly show the issue, higher packet rate and
more likely to have packets in flight at the time of a link
fail...combined with packet loss/retries gives poor throughput.

Depending on what you want to happen, there are a number of tuning
options both on the switches and Linux.  If you want the LAG to be down
if any link fails, the you should be able to config this on the switches
and/or Linux  (minimum number of links = 2 if you have 2 links in the
lag).

You can also tune the link monitoring, how frequently the links are
checked (e.g. miimon) etc.  Bringing this value down from the default of
100ms may allow you to detect a link failure more quickly.  But you then
run into the chance if detecting a transient failure that wouldn’t have
caused any issuesand the LAG becoming more unstable.

Flapping/unstable links are the worst kind of situation.  Ideally you’d
pick that up quickly from monitoring/alerts and either fix immediately
or take the link down until you can fix it.

I run 2x10G from my hosts into separate switches (Dell S series – VLT
between switches).  Pulling a single interface has no impact on Ceph,
any packet loss is tiny and we’re not exceeding 10G bandwidth per host.

If you’re running 1G links and the LAG is already busy, a link failure
could be causing slow writes to the host, just down to
congestion...which then starts to impact the wider cluster based on how
Ceph works.

Just caveating the above with - I’m relatively new to Ceph myself

Sent from 
Mail<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgo.microsoft.com%2Ffwlink%2F%3FLinkId%3D550986data=04%7C01%7C%7Cf6c3c985114b423fb75e08d9309c5248%7C84df9e7fe9f640afb435%7C1%7C0%7C637594270222669937%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=ZrCWPvW06jX69X8PWR0S1tBxhQlzKPQnix4wzBMoN4s%3Dreserved=0>
 for
Windows 10

From: huxia...@horebdata.cn<mailto:huxia...@horebdata.cn>
Sent: 15 June 2021 17:52
To: Serkan Çoban<mailto:cobanser...@gmail.com>
Cc: ceph-users<mailto:ceph-users@ceph.io>
Subject: [ceph-users] Re: Issues with Ceph network redundancy using L2
MC-LAG

When i pull out the cable, then the bond is working properly.

Does it mean that the port is somehow flapping? Ping can still work,
but the iperf test yields very low results.

huxia...@horebdata.cn

From: Serkan Çoban
Date: 2021-06-15 18:47
To: huxia...@horebdata.cn
CC: ceph-users
Subject: Re: [ceph-users] Issues with Ceph network redundancy using L2
MC-LAG
Do you observe the same behaviour when you pull a cable?
Maybe a flapping port might cause this kind of behaviour, other than
that you should't see any network disconnects.
Are you sure about LACP configuration, what is the output of 'cat
/proc/net/bonding/bond0'

On Tue, Jun 15, 2021 at 7:19 PM huxia...@horebdata.cn
 wrote:
> 
> Dear Cephers,
> 
> I encountered the following networking issue several times, and i
wonder whether there is a solution for networking HA solution.
> 
> We build ceph using L2 multi chassis link aggregation group (MC-LAG )
to provide switch redundancy. On each host, we use 802.3ad, LACP
> mode for NIC redundancy. However, we observe several times, when a
single network port, either the cable, or the SFP+ optical module fails,
Ceph cluster  is badly affected by networking, although in theory it
should be able to tolerate.
> 
> Did i miss something important here? and how to really achieve
networking H

[ceph-users] Re: Issues with Ceph network redundancy using L2 MC-LAG

2021-06-15 Thread Andrew Walker-Brown

With an unstable link/port you could see the issues you describe.  Ping doesn’t 
have the packet rate for you to necessarily have a packet in transit at exactly 
the same time as the port fails temporarily.  Iperf on the other hand could 
certainly show the issue, higher packet rate and more likely to have packets in 
flight at the time of a link fail...combined with packet loss/retries gives 
poor throughput.

Depending on what you want to happen, there are a number of tuning options both 
on the switches and Linux.  If you want the LAG to be down if any link fails, 
the you should be able to config this on the switches and/or Linux  (minimum 
number of links = 2 if you have 2 links in the lag).

You can also tune the link monitoring, how frequently the links are checked 
(e.g. miimon) etc.  Bringing this value down from the default of 100ms may 
allow you to detect a link failure more quickly.  But you then run into the 
chance if detecting a transient failure that wouldn’t have caused any 
issuesand the LAG becoming more unstable.

Flapping/unstable links are the worst kind of situation.  Ideally you’d pick 
that up quickly from monitoring/alerts and either fix immediately or take the 
link down until you can fix it.

I run 2x10G from my hosts into separate switches (Dell S series – VLT between 
switches).  Pulling a single interface has no impact on Ceph, any packet loss 
is tiny and we’re not exceeding 10G bandwidth per host.

If you’re running 1G links and the LAG is already busy, a link failure could be 
causing slow writes to the host, just down to congestion...which then starts to 
impact the wider cluster based on how Ceph works.

Just caveating the above with - I’m relatively new to Ceph myself

Sent from Mail for Windows 10

From: huxia...@horebdata.cn
Sent: 15 June 2021 17:52
To: Serkan Çoban
Cc: ceph-users
Subject: [ceph-users] Re: Issues with Ceph network redundancy using L2 MC-LAG

When i pull out the cable, then the bond is working properly.

Does it mean that the port is somehow flapping? Ping can still work, but the 
iperf test yields very low results.

huxia...@horebdata.cn

From: Serkan Çoban
Date: 2021-06-15 18:47
To: huxia...@horebdata.cn
CC: ceph-users
Subject: Re: [ceph-users] Issues with Ceph network redundancy using L2 MC-LAG
Do you observe the same behaviour when you pull a cable?
Maybe a flapping port might cause this kind of behaviour, other than
that you should't see any network disconnects.
Are you sure about LACP configuration, what is the output of 'cat
/proc/net/bonding/bond0'

On Tue, Jun 15, 2021 at 7:19 PM huxia...@horebdata.cn
 wrote:
>
> Dear Cephers,
>
> I encountered the following networking issue several times, and i wonder 
> whether there is a solution for networking HA solution.
>
> We build ceph using L2 multi chassis link aggregation group (MC-LAG ) to 
> provide switch redundancy. On each host, we use 802.3ad, LACP
> mode for NIC redundancy. However, we observe several times, when a single 
> network port, either the cable, or the SFP+ optical module fails, Ceph 
> cluster  is badly affected by networking, although in theory it should be 
> able to tolerate.
>
> Did i miss something important here? and how to really achieve networking HA 
> in Ceph cluster?
>
> best regards,
>
> Samuel
>
>
>
>
> huxia...@horebdata.cn
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

Re: Thermostats for the Blind

2021-06-05 Thread 'Andrew Walker' via VIPhone

I bought my Tado from Amazon in the UK. 

On Friday, June 4, 2021 at 4:57:12 AM UTC+1 arlene...@earthlink.net wrote:

> Where can you buy the tado? it sounds good. Thank you.
>
> - Original Message - 
> *From:* 'Andrew Walker' via VIPhone 
> *To:* VIPhone 
> *Sent:* Wednesday, June 02, 2021 1:58 AM
> *Subject:* Re: Thermostats for the Blind
>
> I have had the Tado thermostat for three or four years now. The app works 
> with Voiceover to set schedules, alter the temperature etc. The Tado can 
> track your movements and turn on or off automatically should nobody be at 
> home. It also works with the Amazon Echo well and may well work with 
> Google, but I have not tried this.
>
> On Tuesday, June 1, 2021 at 9:18:45 PM UTC+1 Mike B wrote:
>
>> Hi All,
>>  
>> I'm in need of a thermostat that can be operated through an app and, is 
>> accessible.  I forget the Honeywell model people have recommended, but I'm 
>> open to other suggestions.  I have the VIP talking thermostat, but I can't 
>> use this since it can't be operated through an app.  This thermostat is for 
>> our 2nd house and this Is why I need to be able to control it from 
>> anywhere.  All help is greatly appreciated.
>>  
>>  
>> Stay safe & take care.  Mike.
>> Just once, I want a username and password prompt to say:  "Close Enough!"
>>  
>>
> -- 
>
> The following information is important for all members of the V iPhone 
> list.
>  
> If you have any questions or concerns about the running of this list, or 
> if you feel that a member's post is inappropriate, please contact the 
> owners or moderators directly rather than posting on the list itself.
>  
> Your V iPhone list moderator is Mark Taylor. Mark can be reached at: 
> mk...@ucla.edu. Your list owner is Cara Quinn - you can reach Cara at 
> cara...@caraquinn.com
>  
> The archives for this list can be searched at:
> http://www.mail-archive.com/vip...@googlegroups.com/ 
> <http://www.mail-archive.com/viphone@googlegroups.com/>
> --- 
> You received this message because you are subscribed to the Google Groups 
> "VIPhone" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to viphone+u...@googlegroups.com.
>
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/viphone/34941139-4af9-4ed3-adc2-242fcb460c9bn%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/viphone/34941139-4af9-4ed3-adc2-242fcb460c9bn%40googlegroups.com?utm_medium=email_source=footer>
> .
>
>

-- 
The following information is important for all members of the V iPhone list.

If you have any questions or concerns about the running of this list, or if you 
feel that a member's post is inappropriate, please contact the owners or 
moderators directly rather than posting on the list itself.

Your V iPhone list moderator is Mark Taylor.  Mark can be reached at:  
mk...@ucla.edu.  Your list owner is Cara Quinn - you can reach Cara at 
caraqu...@caraquinn.com

The archives for this list can be searched at:
http://www.mail-archive.com/viphone@googlegroups.com/
--- 
You received this message because you are subscribed to the Google Groups 
"VIPhone" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to viphone+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/viphone/3257c741-94c5-4f17-bbc5-1944a75571d0n%40googlegroups.com.

[ceph-users] Re: Connect ceph to proxmox

2021-06-05 Thread Andrew Walker-Brown

Hi

Yes you can, check on the Proxmox forums for details, but that’s how I run my 
setup. 

Andrew

Sent from my iPhone

On 5 Jun 2021, at 03:18, Szabo, Istvan (Agoda)  wrote:

Hi,

Is there a way to connect from my nautilus ceph setup the pool that I created 
in ceph to proxmox? Or need a totally different ceph install?

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

Re: Thermostats for the Blind

2021-06-02 Thread 'Andrew Walker' via VIPhone

Hello Mike. I have the Tado thermostat which works well with Voiceover. I 
can set schedules, alter temperatures in the app. It can also be used with 
the Amazon Echo and may be used with Google but I have not tried this. It 
can track your movements and turn on or off if nobody is at home. I have 
had this for three or four years.

On Tuesday, June 1, 2021 at 9:18:45 PM UTC+1 Mike B wrote:

> Hi All,
>  
> I'm in need of a thermostat that can be operated through an app and, is 
> accessible.  I forget the Honeywell model people have recommended, but I'm 
> open to other suggestions.  I have the VIP talking thermostat, but I can't 
> use this since it can't be operated through an app.  This thermostat is for 
> our 2nd house and this Is why I need to be able to control it from 
> anywhere.  All help is greatly appreciated.
>  
>  
> Stay safe & take care.  Mike.
> Just once, I want a username and password prompt to say:  "Close Enough!"
>  
>

-- 
The following information is important for all members of the V iPhone list.

If you have any questions or concerns about the running of this list, or if you 
feel that a member's post is inappropriate, please contact the owners or 
moderators directly rather than posting on the list itself.

Your V iPhone list moderator is Mark Taylor.  Mark can be reached at:  
mk...@ucla.edu.  Your list owner is Cara Quinn - you can reach Cara at 
caraqu...@caraquinn.com

The archives for this list can be searched at:
http://www.mail-archive.com/viphone@googlegroups.com/
--- 
You received this message because you are subscribed to the Google Groups 
"VIPhone" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to viphone+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/viphone/97466d71-cf51-45ef-b5dc-015d72953164n%40googlegroups.com.

Re: Thermostats for the Blind

2021-06-02 Thread 'Andrew Walker' via VIPhone

I have had the Tado thermostat for three or four years now. The app works 
with Voiceover to set schedules, alter the temperature etc. The Tado can 
track your movements and turn on or off automatically should nobody be at 
home. It also works with the Amazon Echo well and may well work with 
Google, but I have not tried this.

On Tuesday, June 1, 2021 at 9:18:45 PM UTC+1 Mike B wrote:

> Hi All,
>  
> I'm in need of a thermostat that can be operated through an app and, is 
> accessible.  I forget the Honeywell model people have recommended, but I'm 
> open to other suggestions.  I have the VIP talking thermostat, but I can't 
> use this since it can't be operated through an app.  This thermostat is for 
> our 2nd house and this Is why I need to be able to control it from 
> anywhere.  All help is greatly appreciated.
>  
>  
> Stay safe & take care.  Mike.
> Just once, I want a username and password prompt to say:  "Close Enough!"
>  
>

-- 
The following information is important for all members of the V iPhone list.

If you have any questions or concerns about the running of this list, or if you 
feel that a member's post is inappropriate, please contact the owners or 
moderators directly rather than posting on the list itself.

Your V iPhone list moderator is Mark Taylor.  Mark can be reached at:  
mk...@ucla.edu.  Your list owner is Cara Quinn - you can reach Cara at 
caraqu...@caraquinn.com

The archives for this list can be searched at:
http://www.mail-archive.com/viphone@googlegroups.com/
--- 
You received this message because you are subscribed to the Google Groups 
"VIPhone" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to viphone+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/viphone/34941139-4af9-4ed3-adc2-242fcb460c9bn%40googlegroups.com.

[ceph-users] Re: OSD lost: firmware bug in Kingston SSDs?

2021-05-13 Thread Andrew Walker-Brown

Hi Frank,

Glad to hear the testing went well and the Kingston SSDs behaved!  Fingers 
crossed your issue was just a corner case...

Cheers,

A.

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

From: Frank Schilder<mailto:fr...@dtu.dk>
Sent: 13 May 2021 10:15
To: Andrew Walker-Brown<mailto:andrew_jbr...@hotmail.com>; 
ceph-users@ceph.io<mailto:ceph-users@ceph.io>
Subject: Re: OSD lost: firmware bug in Kingston SSDs?

Hi Andrew,

I did a few power-out tests  by pulling the power cord of a server several 
times. This server contains a mix of disks, including the Kingston SSDs (also 
the one that failed before). Every time, all OSDs recovered and an initiated 
deep scrub did not find silent corruptions either. The test was done under 
production load.

Looks like the OSD crash I observed was caused by special and hopefully rare 
circumstances.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Frank Schilder 
Sent: 06 May 2021 15:27:14
To: Andrew Walker-Brown; ceph-users@ceph.io
Subject: [ceph-users] Re: OSD lost: firmware bug in Kingston SSDs?

Hi Andrew,

thanks, that is reassuring. To be sure, I plan to do a few power out tests with 
this server. Never had any issues with that so far, its the first time I see a 
corrupted OSD.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14

________
From: Andrew Walker-Brown 
Sent: 06 May 2021 15:23:30
To: Frank Schilder; ceph-users@ceph.io
Subject: RE: OSD lost: firmware bug in Kingston SSDs?

Hi Frank,

I’m running the same SSDs (approx. 20) in Dell servers on HBA330’s.  Haven’t 
had any issues and have suffered at least one power outage.  Just checking the 
wcache setting and it shows as enabled.

Running Octopus 15.1.9 and docker containers.  Originally part of a Proxmox 
cluster but now standalone Ceph.

Cheers,

A

Sent from 
Mail<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgo.microsoft.com%2Ffwlink%2F%3FLinkId%3D550986data=04%7C01%7C%7C4ab9432f42c849ff166208d915efb10d%7C84df9e7fe9f640afb435%7C1%7C0%7C637564941470092873%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=kgh98gPGirs9BFdSM5n6T3F9UhXUiux3aU1ScITUEf4%3Dreserved=0>
 for Windows 10

From: Frank Schilder<mailto:fr...@dtu.dk>
Sent: 06 May 2021 10:11
To: ceph-users@ceph.io<mailto:ceph-users@ceph.io>
Subject: [ceph-users] OSD lost: firmware bug in Kingston SSDs?

Hi all,

I lost 2 OSDs deployed on a single Kingston SSD in a rather strange way and am 
wondering if anyone has made similar observations or is aware of a firmware bug 
with these disks.

Disk model: KINGSTON SEDC500M3840G (it ought to be a DC grade model with super 
capacitors)
Smartctl does not report any drive errors.
Performance per TB is as expected, OSDs are "ceph-volume lvm batch" bluestore 
deployed, everything collocated.

Short version: I disable volatile write cache on all OSD disks, but the 
Kingston disks seem to behave as if this cache is *not* disabled. Smartctl and 
hdparm report wcache=off though. The OSD loss looks like what unflushed write 
cache during power loss would result in. I'm afraid now that our cluster might 
be vulnerable to power loss.

Long version:

Our disks are on Dell HBA330 Mini controllers and are in state "non-raid". The 
controller itself has no cache and is HBA-mode only.

Log entry:

The iDRAC log shows that the disk was removed from a drive group:

---
PDR5 Disk 6 in Backplane 2 of Integrated Storage Controller 1 is removed.
Detailed Description: A physical disk has been removed from the disk group. 
This alert can also be caused by loose or defective cables or by problems with 
the enclosure.
---

The iDRAC did not report the disk as failed and neither as "removed from drive 
bay". I reseated the disk and it came back as healthy. I assume it was a 
problem with connectivity to the back-plane (chassis). If I now try to start up 
the OSDs on this disk, I get the error:

starting osd.581 at - osd_data /var/lib/ceph/osd/ceph-581 
/var/lib/ceph/osd/ceph-581/journal
starting osd.580 at - osd_data /var/lib/ceph/osd/ceph-580 
/var/lib/ceph/osd/ceph-580/journal
2021-05-06 09:23:47.160 7fead5a1fb80 -1 bluefs mount failed to replay log: (5) 
Input/output error
2021-05-06 09:23:47.160 7fead5a1fb80 -1 bluestore(/var/lib/ceph/osd/ceph-581) 
_open_db failed bluefs mount: (5) Input/output error
2021-05-06 09:23:47.630 7fead5a1fb80 -1 osd.581 0 OSD:init: unable to mount 
object store
2021-05-06 09:23:47.630 7fead5a1fb80 -1  ** ERROR: osd init failed: (5) 
Input/output error

I have removed disks of active OSDs before without any bluestore corruption 
happening. While it is very well possible that this particular "disconnect" 
event may lead to a broken OSD, there is also an

[ceph-users] Re: OSD lost: firmware bug in Kingston SSDs?

2021-05-06 Thread Andrew Walker-Brown

Hi Frank,

I’m running the same SSDs (approx. 20) in Dell servers on HBA330’s.  Haven’t 
had any issues and have suffered at least one power outage.  Just checking the 
wcache setting and it shows as enabled.

Running Octopus 15.1.9 and docker containers.  Originally part of a Proxmox 
cluster but now standalone Ceph.

Cheers,

A

Sent from Mail for Windows 10

From: Frank Schilder
Sent: 06 May 2021 10:11
To: ceph-users@ceph.io
Subject: [ceph-users] OSD lost: firmware bug in Kingston SSDs?

Hi all,

I lost 2 OSDs deployed on a single Kingston SSD in a rather strange way and am 
wondering if anyone has made similar observations or is aware of a firmware bug 
with these disks.

Disk model: KINGSTON SEDC500M3840G (it ought to be a DC grade model with super 
capacitors)
Smartctl does not report any drive errors.
Performance per TB is as expected, OSDs are "ceph-volume lvm batch" bluestore 
deployed, everything collocated.

Short version: I disable volatile write cache on all OSD disks, but the 
Kingston disks seem to behave as if this cache is *not* disabled. Smartctl and 
hdparm report wcache=off though. The OSD loss looks like what unflushed write 
cache during power loss would result in. I'm afraid now that our cluster might 
be vulnerable to power loss.

Long version:

Our disks are on Dell HBA330 Mini controllers and are in state "non-raid". The 
controller itself has no cache and is HBA-mode only.

Log entry:

The iDRAC log shows that the disk was removed from a drive group:

---
PDR5 Disk 6 in Backplane 2 of Integrated Storage Controller 1 is removed.
Detailed Description: A physical disk has been removed from the disk group. 
This alert can also be caused by loose or defective cables or by problems with 
the enclosure.
---

The iDRAC did not report the disk as failed and neither as "removed from drive 
bay". I reseated the disk and it came back as healthy. I assume it was a 
problem with connectivity to the back-plane (chassis). If I now try to start up 
the OSDs on this disk, I get the error:

starting osd.581 at - osd_data /var/lib/ceph/osd/ceph-581 
/var/lib/ceph/osd/ceph-581/journal
starting osd.580 at - osd_data /var/lib/ceph/osd/ceph-580 
/var/lib/ceph/osd/ceph-580/journal
2021-05-06 09:23:47.160 7fead5a1fb80 -1 bluefs mount failed to replay log: (5) 
Input/output error
2021-05-06 09:23:47.160 7fead5a1fb80 -1 bluestore(/var/lib/ceph/osd/ceph-581) 
_open_db failed bluefs mount: (5) Input/output error
2021-05-06 09:23:47.630 7fead5a1fb80 -1 osd.581 0 OSD:init: unable to mount 
object store
2021-05-06 09:23:47.630 7fead5a1fb80 -1  ** ERROR: osd init failed: (5) 
Input/output error

I have removed disks of active OSDs before without any bluestore corruption 
happening. While it is very well possible that this particular "disconnect" 
event may lead to a broken OSD, there is also another observation where the 
Kingston disks stick out compared with other SSD OSDs, which make me suspicious 
of this being a disk cache firmware problem:

The I/O indicator LED lights up with significantly lower frequency than for all 
other SSD types on the same pool even though we have 2 instead of 1 OSD 
deployed on the Kingstons (the other disks are 2TB Micron Pro). While this 
could be due to a wiring difference I'm starting to suspect that this might be 
an indication of volatile caching.

Does anyone using Kingston DC-M-SSDs have similar or contradicting experience?
How did these disks handle power outages?
Any recommendations?

Thanks and best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Logging to Graylog

2021-04-19 Thread Andrew Walker-Brown

Hi All,

I want to send Ceph logs out to an external Graylog server.  I’ve configured 
the Graylog host IP using “ceph config set global log_graylog_host x.x.x.x” and 
enabled logging through the Ceph dashboard (I’m running Octopus 15.2.9 – 
container based).  I’ve also setup a GELF UDP input on Graylog.

Do I need to do anything else?  Any services that need to be restarted?

Cheers

Andrew

Sent from Mail for Windows 10
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: HEALTH_WARN - Recovery Stuck?

2021-04-12 Thread Andrew Walker-Brown

If you increase the number of pgs, effectively each one is smaller so the 
backfill process may be able to ‘squeeze’ them onto the nearly full osds while 
it sorts things out. 

I’ve had something similar before and this def helped. 

Sent from my iPhone

On 12 Apr 2021, at 19:11, Marc  wrote:


You know you can play a bit with the ratios?

ceph tell osd.* injectargs '--mon_osd_full_ratio=0.95'
ceph tell osd.* injectargs '--mon_osd_backfillfull_ratio=0.90'


> -Original Message-
> From: Ml Ml 
> Sent: 12 April 2021 19:31
> To: ceph-users 
> Subject: [ceph-users] HEALTH_WARN - Recovery Stuck?
> 
> Hello,
> 
> i kind of ran out of disk space, so i added another host with osd.37.
> But it does not seem to move much data on it. (85MB in 2h)
> 
> Any idea why the recovery process seems to be stuck? Should i fix the
> 4 backfillfull osds first? (by changing the weight)?
> 
> root@ceph01:~# ceph -s
>  cluster:
>id: 5436dd5d-83d4-4dc8-a93b-60ab5db145df
>health: HEALTH_WARN
>4 backfillfull osd(s)
>9 nearfull osd(s)
>Low space hindering backfill (add storage if this doesn't
> resolve itself): 1 pg backfill_toofull
>4 pool(s) backfillfull
> 
>  services:
>mon: 3 daemons, quorum ceph03,ceph01,ceph02 (age 12d)
>mgr: ceph03(active, since 4M), standbys: ceph02.jwvivm
>mds: backup:1 {0=backup.ceph06.hdjehi=up:active} 3 up:standby
>osd: 53 osds: 53 up (since 2h), 53 in (since 2h); 235 remapped pgs
> 
>  task status:
>scrub status:
>mds.backup.ceph06.hdjehi: idle
> 
>  data:
>pools:   4 pools, 1185 pgs
>objects: 24.69M objects, 45 TiB
>usage:   149 TiB used, 42 TiB / 191 TiB avail
>pgs: 5388809/74059569 objects misplaced (7.276%)
> 950 active+clean
> 232 active+remapped+backfill_wait
> 2   active+remapped+backfilling
> 1   active+remapped+backfill_wait+backfill_toofull
> 
>  io:
>recovery: 0 B/s, 171 keys/s, 16 objects/s
> 
>  progress:
>Rebalancing after osd.37 marked in (2h)
>  [] (remaining: 6d)
> 
> 
> 
> root@ceph01:~# ceph health detail
> HEALTH_WARN 4 backfillfull osd(s); 9 nearfull osd(s); Low space
> hindering backfill (add storage if this doesn't resolve itself): 1 pg
> backfill_toofull; 4 pool(s) backfillfull
> [WRN] OSD_BACKFILLFULL: 4 backfillfull osd(s)
>osd.28 is backfill full
>osd.32 is backfill full
>osd.66 is backfill full
>osd.68 is backfill full
> [WRN] OSD_NEARFULL: 9 nearfull osd(s)
>osd.11 is near full
>osd.24 is near full
>osd.27 is near full
>osd.39 is near full
>osd.40 is near full
>osd.42 is near full
>osd.43 is near full
>osd.45 is near full
>osd.69 is near full
> [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if
> this doesn't resolve itself): 1 pg backfill_toofull
>pg 23.295 is active+remapped+backfill_wait+backfill_toofull,
> acting [8,67,32]
> [WRN] POOL_BACKFILLFULL: 4 pool(s) backfillfull
>pool 'backurne-rbd' is backfillfull
>pool 'device_health_metrics' is backfillfull
>pool 'cephfs.backup.meta' is backfillfull
>pool 'cephfs.backup.data' is backfillfull
> 
> 
> root@ceph01:~# ceph osd df tree
> ID   CLASS  WEIGHT REWEIGHT  SIZE RAW USE  DATA OMAP
> META AVAIL%USE   VAR   PGS  STATUS  TYPE NAME
> -1 182.59897 -  191 TiB  149 TiB  149 TiB35 GiB
> 503 GiB   42 TiB  77.96  1.00-  root default
> -2  24.62473 -   29 TiB   22 TiB   22 TiB   5.0 GiB
> 80 GiB  7.1 TiB  75.23  0.96-  host ceph01
>  0hdd2.3   1.0  2.7 TiB  2.2 TiB  2.2 TiB   665 MiB
> 8.0 GiB  480 GiB  82.43  1.06   53  up  osd.0
>  1hdd2.2   1.0  2.7 TiB  2.1 TiB  2.1 TiB   446 MiB
> 7.5 GiB  590 GiB  78.44  1.01   49  up  osd.1
>  4hdd2.67029   0.91066  2.7 TiB  2.2 TiB  2.2 TiB   484 MiB
> 7.9 GiB  440 GiB  83.90  1.08   53  up  osd.4
>  8hdd2.3   1.0  2.7 TiB  2.1 TiB  2.1 TiB   490 MiB
> 7.9 GiB  533 GiB  80.49  1.03   51  up  osd.8
> 11hdd1.71660   1.0  1.7 TiB  1.5 TiB  1.5 TiB   406 MiB
> 5.5 GiB  200 GiB  88.60  1.14   36  up  osd.11
> 12hdd1.2   1.0  2.7 TiB  1.2 TiB  1.2 TiB   366 MiB
> 4.9 GiB  1.5 TiB  43.89  0.56   28  up  osd.12
> 14hdd2.2   1.0  2.7 TiB  2.0 TiB  2.0 TiB   418 MiB
> 7.1 GiB  693 GiB  74.66  0.96   47  up  osd.14
> 18hdd2.2   1.0  2.7 TiB  2.0 TiB  1.9 TiB   434 MiB
> 7.3 GiB  737 GiB  73.05  0.94   47  up  osd.18
> 22hdd1.0   1.0  1.7 TiB  890 GiB  886 GiB   110 MiB
> 3.6 GiB  868 GiB  50.62  0.65   20  up  osd.22
> 30hdd1.5   1.0  1.7 TiB  1.4 TiB  1.3 TiB   361 MiB
> 4.9 GiB  370 GiB  78.93  1.01   32  up  osd.30
> 33hdd1.5   0.97437  1.6 TiB

[ceph-users] Re: Upgrade and lost osds Operation not permitted

And after a reboot what errors are you getting?

Sent from my iPhone

On 4 Apr 2021, at 15:33, Behzad Khoshbakhti  wrote:


I have changed the uid and gid to 167, but still no progress.
cat /etc/group | grep -i ceph
ceph:x:167:
root@osd03:~# cat /etc/passwd | grep -i ceph
ceph:x:167:167:Ceph storage service:/var/lib/ceph:/usr/sbin/nologin

On Sun, Apr 4, 2021 at 6:47 PM Andrew Walker-Brown 
mailto:andrew_jbr...@hotmail.com>> wrote:
UID and guid should both be 167 I believe.

Make a note of the current values and change them to 167 using usermod and 
groupmod.

I had just this issue. It’s partly to do with how perms are used within the 
containers I think.

I changed the values to 167 in passwd everything worked again. Symptoms for me 
were OSDs not starting and permissions/file not found errors.

Sent from my iPhone

On 4 Apr 2021, at 13:43, Lomayani S. Laizer 
mailto:lomlai...@gmail.com>> wrote:


Hello,
Permissions are correct. guid/uid is 64045/64045

ls -alh
total 32K
drwxrwxrwt 2 ceph ceph  200 Apr  4 14:11 .
drwxr-xr-x 8 ceph ceph 4.0K Sep 18  2018 ..
lrwxrwxrwx 1 ceph ceph   93 Apr  4 14:11 block -> /dev/...
-rw--- 1 ceph ceph   37 Apr  4 14:11 ceph_fsid
-rw--- 1 ceph ceph   37 Apr  4 14:11 fsid
-rw--- 1 ceph ceph   56 Apr  4 14:11 keyring
-rw--- 1 ceph ceph6 Apr  4 14:11 ready
-rw--- 1 ceph ceph3 Apr  4 14:11 require_osd_release
-rw--- 1 ceph ceph   10 Apr  4 14:11 type
-rw--- 1 ceph ceph3 Apr  4 14:11 whoami

On Sun, Apr 4, 2021 at 3:07 PM Andrew Walker-Brown 
mailto:andrew_jbr...@hotmail.com>> wrote:
Are the file permissions correct and UID/guid in passwd  both 167?

Sent from my iPhone

On 4 Apr 2021, at 12:29, Lomayani S. Laizer 
mailto:lomlai...@gmail.com>> wrote:

Hello,

+1 Am facing the same problem in ubuntu after upgrade to pacific

2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 bluestore(/var/lib/ceph/osd/
ceph-29/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-29/block:
(1) Operation not permitted
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable to
open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or
directoryESC[0m

On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti 
mailto:khoshbakh...@gmail.com>>
wrote:

> It worth mentioning as I issue the following command, the Ceph OSD starts
> and joins the cluster:
> /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
>
>
>
> On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti 
> mailto:khoshbakh...@gmail.com>>
> wrote:
>
>> Hi all,
>>
>> As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the
>> manual upgrade using the precompiled packages, the OSDs was down with the
>> following messages:
>>
>> root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all
>> --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352
>> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
>> Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph
> prime-osd-dir
>> --dev
>>
> /dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
>> --path /var/lib/ceph/osd/ceph-2 --no-mon-config
>> Running command: /usr/bin/ln -snf
>>
> /dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
>> /var/lib/ceph/osd/ceph-2/block
>> Running command: /usr/bin/chown -h ceph:ceph
> /var/lib/ceph/osd/ceph-2/block
>> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1
>> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
>> Running command: /usr/bin/systemctl enable
>> ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352
>> Running command: /usr/bin/systemctl enable --runtime ceph-osd@2
>> Running command: /usr/bin/systemctl start ceph-osd@2
>> --> ceph-volume lvm activate successful for osd ID: 2
>>
>> Content of /var/log/ceph/ceph-osd.2.log
>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00  0 set uid:gid to 64045:64045
>> (ceph:ceph)
>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00  0 ceph version 16.2.0
>> (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process
>> ceph-osd, pid 5484
>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00  0 pidfile_write: ignore empty
>> --pid-file
>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1*
>> bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open
>> /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted*
>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1  *** ERROR: unable to open
>> OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or
> directory*
>>
>>
>> root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2
>&g

[ceph-users] Re: Upgrade and lost osds Operation not permitted

UID and guid should both be 167 I believe.

Make a note of the current values and change them to 167 using usermod and 
groupmod.

I had just this issue. It’s partly to do with how perms are used within the 
containers I think.

I changed the values to 167 in passwd everything worked again. Symptoms for me 
were OSDs not starting and permissions/file not found errors.

Sent from my iPhone

On 4 Apr 2021, at 13:43, Lomayani S. Laizer  wrote:


Hello,
Permissions are correct. guid/uid is 64045/64045

ls -alh
total 32K
drwxrwxrwt 2 ceph ceph  200 Apr  4 14:11 .
drwxr-xr-x 8 ceph ceph 4.0K Sep 18  2018 ..
lrwxrwxrwx 1 ceph ceph   93 Apr  4 14:11 block -> /dev/...
-rw--- 1 ceph ceph   37 Apr  4 14:11 ceph_fsid
-rw--- 1 ceph ceph   37 Apr  4 14:11 fsid
-rw--- 1 ceph ceph   56 Apr  4 14:11 keyring
-rw--- 1 ceph ceph6 Apr  4 14:11 ready
-rw--- 1 ceph ceph3 Apr  4 14:11 require_osd_release
-rw--- 1 ceph ceph   10 Apr  4 14:11 type
-rw--- 1 ceph ceph3 Apr  4 14:11 whoami

On Sun, Apr 4, 2021 at 3:07 PM Andrew Walker-Brown 
mailto:andrew_jbr...@hotmail.com>> wrote:
Are the file permissions correct and UID/guid in passwd  both 167?

Sent from my iPhone

On 4 Apr 2021, at 12:29, Lomayani S. Laizer 
mailto:lomlai...@gmail.com>> wrote:

Hello,

+1 Am facing the same problem in ubuntu after upgrade to pacific

2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 bluestore(/var/lib/ceph/osd/
ceph-29/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-29/block:
(1) Operation not permitted
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable to
open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or
directoryESC[0m

On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti 
mailto:khoshbakh...@gmail.com>>
wrote:

> It worth mentioning as I issue the following command, the Ceph OSD starts
> and joins the cluster:
> /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
>
>
>
> On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti 
> mailto:khoshbakh...@gmail.com>>
> wrote:
>
>> Hi all,
>>
>> As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the
>> manual upgrade using the precompiled packages, the OSDs was down with the
>> following messages:
>>
>> root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all
>> --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352
>> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
>> Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph
> prime-osd-dir
>> --dev
>>
> /dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
>> --path /var/lib/ceph/osd/ceph-2 --no-mon-config
>> Running command: /usr/bin/ln -snf
>>
> /dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
>> /var/lib/ceph/osd/ceph-2/block
>> Running command: /usr/bin/chown -h ceph:ceph
> /var/lib/ceph/osd/ceph-2/block
>> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1
>> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
>> Running command: /usr/bin/systemctl enable
>> ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352
>> Running command: /usr/bin/systemctl enable --runtime ceph-osd@2
>> Running command: /usr/bin/systemctl start ceph-osd@2
>> --> ceph-volume lvm activate successful for osd ID: 2
>>
>> Content of /var/log/ceph/ceph-osd.2.log
>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00  0 set uid:gid to 64045:64045
>> (ceph:ceph)
>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00  0 ceph version 16.2.0
>> (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process
>> ceph-osd, pid 5484
>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00  0 pidfile_write: ignore empty
>> --pid-file
>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1*
>> bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open
>> /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted*
>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1  *** ERROR: unable to open
>> OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or
> directory*
>>
>>
>> root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2
>> â— ceph-osd@2.service - Ceph object storage daemon osd.2
>> Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled;
>> vendor preset: enabled)
>> Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06
>> +0430; 50s ago
>>Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh
>> --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS)
>>Process: 5484 ExecStart=/usr/bin/ceph-osd -f --

[ceph-users] Re: Upgrade and lost osds Operation not permitted

Are the file permissions correct and UID/guid in passwd  both 167?

Sent from my iPhone

On 4 Apr 2021, at 12:29, Lomayani S. Laizer  wrote:

Hello,

+1 Am facing the same problem in ubuntu after upgrade to pacific

2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 bluestore(/var/lib/ceph/osd/
ceph-29/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-29/block:
(1) Operation not permitted
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable to
open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or
directoryESC[0m

On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti 
wrote:

> It worth mentioning as I issue the following command, the Ceph OSD starts
> and joins the cluster:
> /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
> 
> 
> 
> On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti 
> wrote:
> 
>> Hi all,
>> 
>> As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the
>> manual upgrade using the precompiled packages, the OSDs was down with the
>> following messages:
>> 
>> root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all
>> --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352
>> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
>> Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph
> prime-osd-dir
>> --dev
>> 
> /dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
>> --path /var/lib/ceph/osd/ceph-2 --no-mon-config
>> Running command: /usr/bin/ln -snf
>> 
> /dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
>> /var/lib/ceph/osd/ceph-2/block
>> Running command: /usr/bin/chown -h ceph:ceph
> /var/lib/ceph/osd/ceph-2/block
>> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1
>> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
>> Running command: /usr/bin/systemctl enable
>> ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352
>> Running command: /usr/bin/systemctl enable --runtime ceph-osd@2
>> Running command: /usr/bin/systemctl start ceph-osd@2
>> --> ceph-volume lvm activate successful for osd ID: 2
>> 
>> Content of /var/log/ceph/ceph-osd.2.log
>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00  0 set uid:gid to 64045:64045
>> (ceph:ceph)
>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00  0 ceph version 16.2.0
>> (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process
>> ceph-osd, pid 5484
>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00  0 pidfile_write: ignore empty
>> --pid-file
>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1*
>> bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open
>> /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted*
>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1  *** ERROR: unable to open
>> OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or
> directory*
>> 
>> 
>> root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2
>> â— ceph-osd@2.service - Ceph object storage daemon osd.2
>> Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled;
>> vendor preset: enabled)
>> Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06
>> +0430; 50s ago
>>Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh
>> --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS)
>>Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER}
> --id
>> 2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
>>   Main PID: 5484 (code=exited, status=1/FAILURE)
>> 
>> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Scheduled restart
>> job, restart counter is at 3.
>> Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon
> osd.2.
>> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Start request
>> repeated too quickly.
>> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Failed with result
>> 'exit-code'.
>> Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage
>> daemon osd.2.
>> root@osd03:/var/lib/ceph/osd/ceph-2#
>> 
>> root@osd03:~# lsblk
>> NAME  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
>> fd0 2:014K  0 disk
>> loop0   7:00 55.5M  1 loop
>> /snap/core18/1988
>> loop1   7:10 69.9M  1 loop
>> /snap/lxd/19188
>> loop2   7:20 55.5M  1 loop
>> /snap/core18/1997
>> loop3   7:30 70.4M  1 loop
>> /snap/lxd/19647
>> loop4   7:40 32.3M  1 loop
>> /snap/snapd/11402
>> loop5   7:50 32.3M  1 loop
>> /snap/snapd/11107
>> sda 8:00   80G  0 disk
>> ├─sda1  8:101M  0 part
>> ├─sda2  8:201G  0 part /boot
>> └─sda3

[ceph-users] Re: Installation of Ceph on Ubuntu 18.04 TLS

Majid,

Check out the install guide for Ubuntu using Cephadm here:

https://docs.ceph.com/en/latest/cephadm/install/

Basically install cephadm using apt, then use the Cephadm bootstrap command to
get the first mon up and running.

For additional hosts, make sure you have them in the hosts file and also make
sure you do this step, adding the Ceph ssh keys, before attempting to add new
hosts:

https://docs.ceph.com/en/latest/cephadm/host-management/#cephadm-adding-hosts

Once the first mon/mgr is up (after the bootstrap) you can do most of what you
need via the web interface, adding hosts/osd’s etc.

You will have fun and games with it, the email list and iirc channels are very
active and helpful.

Just be aware of the many different deployment methods of Ceph e.g. Cephadm,
ansible, juju etc. Also whether native or container based install. The
direction of travel appears to be with Cephadm and containers. The
documentation you’ll find out on the web may be referring to a different
deployment method or tools compared to what you’ve done. Even the official docs
can be a bit misleading sometimes.

I’m pretty new to Ceph myself and deployed onto Ubuntu 20.04 LTS and Ceph
15.2.8/9 on containers after using Ceph on a Proxmox cluster. It’s in
production now and seems to work well.

Break it and fix it a few timesyou’ll get the hang of it.

Cheers

Sent from my iPhone

On 4 Apr 2021, at 10:13, Michel Niyoyita wrote:

Thanks Dear Madjid,

Have you successfully installed ceph? If yes Please help me , I beg you.
on that link , I tried to open /etc/hosts/ and insert IP , but i did not
succeed to save.

On Sun, Apr 4, 2021 at 10:53 AM Majid Varzideh wrote:

hi
the best is looking at the ceph documentations.
in meantime you can use this link:

https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcomputingforgeeks.com%2Fhow-to-deploy-ceph-storage-cluster-on-ubuntu-18-04-lts%2Fdata=04%7C01%7C%7C4ece5ae17df046f0ce3f08d8f749dea2%7C84df9e7fe9f640afb435%7C1%7C0%7C637531243926365905%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=Jw2ED%2B4Ewdzjlq%2BWTul4az9CND%2BJywqY8Ypb43H3Amc%3Dreserved=0

On Sun, Apr 4, 2021 at 12:54 PM Michel Niyoyita wrote:

Dear Ceph users,

Kindly help on how I can deploy ceph on ubuntu 18.04 TLS , I am learning
Ceph from scratch , please your inputs are highly appreciated.

Regards
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

Re: [go-nuts] Transferring host's go-mod-download cache into docker container

2021-03-31 Thread Andrew Walker

There are a couple ways I know of that you can do this. You can vendor or 
copy as you mention.  Another approach is to create a "build" image that 
contains relevant credentials and a few `go env -w` directives to set up 
your build environment, which you build separately and derive other 
projects from.

```
FROM golang:1.16-alpine3.13
RUN apk --no-cache add ca-certificates git openssh
# cert not checked in to repo, (download with artifact script to rebuild 
base images)
ADD certs/mycert.crt /usr/local/share/ca-certificates/mycert.crt
RUN chmod 0600 /usr/local/share/ca-certificates/mycert.crt
RUN update-ca-certificates

# if you host a proxy, use these two lines or similar (see 
https://golang.org/ref/mod#private-modules)
# RUN go env -w GOPROXY=https://proxy.corp.example.com
# RUN go env -w GONOSUMDB=corp.example.com

# otherwise, make sure you don't reach out to public proxy/sum for internal 
packages
RUN go env -w GOPRIVATE=vcs.package.prefix/*

# gitconfig for vcs to use "ssh://git@" instead of "https://;
ADD gitconfig /root/.gitconfig
# internal VCS creds (probably want to use a "deployment key")
ADD ssh /root/.ssh
RUN chmod 0600 /root/.ssh/id_rsa /root/.ssh/config
RUN chmod 0644 /root/.ssh/id_rsa.pub
```

And then the gitconfig you copy in might be something like:

```
[url "ssh://git@my.internal.bitbucket:7999"]
insteadOf = "https://my.internal.bitbucket/scm;
```

This should create a build image that will know about your VCS and download 
modules from it directly.  In your project, you can either do the standard 
copy of your mod and sum files and go download, but you can also use the 
new buildkit features to mount a shared cache for both modules and 
artifacts:

```
# syntax = docker/dockerfile:1-experimental
FROM myregistry:base as base
# ...
RUN --mount=type=cache,id=myproject_mod,target=/go/pkg/mod \
--mount=type=cache,id=myproject_cache,target=/root/.cache/go-build \
go build
```
On Tuesday, January 15, 2019 at 11:01:02 AM UTC-5 greg...@unity3d.com wrote:

> On Monday, January 14, 2019 at 4:52:37 PM UTC-8, Philip Nelson wrote:
>>
>> You can't copy from outside the context but you can volume mount it. In 
>> my docker-compose.yml I have the following for a golang:1.11-alpine image 
>> derivative:
>>
>> ```
>> volumes:
>>   - ${GOPATH}:/go
>> ```
>>
>> Phil
>>
>>>
>>>
> Oh! That's interesting. Thanks!
>  
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/4b13ada3-362d-4075-9309-0e59e2595fd4n%40googlegroups.com.

[ceph-users] Re: howto:: emergency shutdown procedure and maintenance

2021-03-19 Thread Andrew Walker-Brown

Hi Adrian,

For maintenance, this is the procedure I’d follow:

https://ceph.io/planet/how-to-do-a-ceph-cluster-maintenance-shutdown/

Difference between maintenance and emergency; I’d probably set all the flags as 
per maintenance but down the OSD’s at the same time followed by all the 
mons/admin.

If you scripted it, there’s probably not a lot of difference in the time take 
to shutdown.

A.

Sent from Mail for Windows 10

From: Adrian Sevcenco
Sent: 18 March 2021 19:26
To: ceph-users
Subject: [ceph-users] howto:: emergency shutdown procedure and maintenance

Hi! What steps/procedures are required for emergency shutdown and for machine 
maintenance?

Thanks a lot!
Adrian


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Same data for two buildings

Denis,

I’m doing something similar to you with 5 nodes, 4 with OSDs and a 5th just as 
a mon. 

I have pools set with 4 replicas, minimum 2, crush map configured so 2 replicas 
go to each DC and then to host level. 

5th mon is in a third location, but could be a VM with higher latency somewhere 
else.   It’s primary purpose is to provide quorum if I lose a DC. 

With this setup I can lose a full DC and still have a working cluster. 

The Pacific release has some optimisations around this ‘stretched cluster’ 
deployment. In Pacific, OSDs will only communicate with mon in same DC and In 
event of DC loss then pool min_size automatically reduces to 1. Mon election 
also has some optimisations which help in asymmetric network failures. 

There’s a good youtube video from early 2020 which talks about stretched 
cluster deployment and it’s actually in the development version admin documents 
 on Ceph.io

Regards

A

Sent from my iPhone

On 17 Mar 2021, at 18:08, Denis Morejon Lopez  wrote:



⁣I have a ceph cluster with 5 nodes. 3 in one building, and 2 in the other one. 
I put this information in the CRUSH. So that ceph were able to put one copy of 
objects in the nodes of one building and the other copy to the nodes of the 
other building. I mean, I setup replicas=2 in order to put the same information 
in both locations. But I know ceph cluster needs half +1 up nodes to keep one 
part of cluster working. I need at least a manual procedure to recover one of 
the two buildings if the other get down, or even if the link between them get 
down. I do not need 100% up, just something to block and unblock some nodes, 
and starts the two nodes if the building down were the one with 3 nodes.

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Email alerts from Ceph

Hi all,

How have folks implemented getting email or snmp alerts out of Ceph?  Getting 
things like osd/pool nearly full or osd/daemon failures etc. 

Kind regards

Andrew

Sent from my iPhone
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Quick quota question

Ahh ok, good to know!

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

From: Stefan Kooman<mailto:ste...@bit.nl>
Sent: 17 March 2021 10:37
To: Andrew Walker-Brown<mailto:andrew_jbr...@hotmail.com>; Magnus 
HAGDORN<mailto:magnus.hagd...@ed.ac.uk>; 
ceph-users@ceph.io<mailto:ceph-users@ceph.io>
Subject: Re: [ceph-users] Re: Quick quota question

On 3/17/21 11:28 AM, Andrew Walker-Brown wrote:
> Hi Magnus,
>
> Thanks for the reply.  Just to be certain (I’m having a slow day today), it’s 
> the amount of data stored by the clients.  As an example. a pool using 3 
> replicas and a quota 3TB : clients would be able to create up to 3TB of data 
> and Ceph would use 9TB of raw storage?

Yes, but remember it's "client side security". So if the client does not
play well nicely (malicious intent / buggy), it can ignore quotas. So
make sure you have checks in place so you notice this. Of course depends
on the threat model you have.

FYI,

Gr. Stefan

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Quick quota question

Thank you!

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

From: Burkhard Linke<mailto:burkhard.li...@computational.bio.uni-giessen.de>
Sent: 17 March 2021 10:32
To: ceph-users@ceph.io<mailto:ceph-users@ceph.io>
Subject: [ceph-users] Re: Quick quota question

Hi,

On 3/17/21 11:28 AM, Andrew Walker-Brown wrote:
> Hi Magnus,
>
> Thanks for the reply.  Just to be certain (I’m having a slow day today), it’s 
> the amount of data stored by the clients.  As an example. a pool using 3 
> replicas and a quota 3TB : clients would be able to create up to 3TB of data 
> and Ceph would use 9TB of raw storage?


In case of replicating pools with size=3, yes.


Quotas are enforced by the cephfs clients (either kernel or fuse). And
the clients do not know about the pool details (replicated vs. EC, sizes
etc.). They only care for the primary OSD for the PGs (roughly
speaking), since this OSD is handling the replication / EC stuff.


Regards,

burkhard

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Quick quota question

Hi Magnus,

Thanks for the reply.  Just to be certain (I’m having a slow day today), it’s 
the amount of data stored by the clients.  As an example. a pool using 3 
replicas and a quota 3TB : clients would be able to create up to 3TB of data 
and Ceph would use 9TB of raw storage?

Cheers,

A.

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

From: Magnus HAGDORN<mailto:magnus.hagd...@ed.ac.uk>
Sent: 17 March 2021 08:59
To: ceph-users@ceph.io<mailto:ceph-users@ceph.io>
Subject: [ceph-users] Re: Quick quota question

On Wed, 2021-03-17 at 08:26 +0000, Andrew Walker-Brown wrote:
> When setting a quota on a pool (or directory in Cephfs), is it the
> amount of client data written or the client data x number of replicas
> that counts toward the quota?

It's the amount of data stored so independent of replication level.
The University of Edinburgh is a charitable body, registered in Scotland, with 
registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh 
Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Quick quota question