date:20161006

Re: [ceph-users] creating a rbd

2016-10-06 Thread Jaemyoun Lee

Thanks a lot.

On Fri, Oct 7, 2016 at 2:19 PM Daleep Singh Bais 
wrote:

> Hi,
>
> Ceph uses other ports also for other daemons like OSD and MDS. Please
> refer to
>
> http://docs.ceph.com/docs/jewel/rados/configuration/network-config-ref/
>
> This might help you to resolve the issue.
>
> Thanks.
>
>
> Daleep Singh Bais
>
>
> On 10/07/2016 10:14 AM, Jaemyoun Lee wrote:
>
> Dear Laizer,
>
> Oh, I got it. the rbd was created successfully after I disabled the
> firewall.
> However, when the firewall is enabled and the 6789 port is allowed, the
> authentication error is occurred.
>
> Is there any other port?
>
> On Fri, Oct 7, 2016 at 1:23 PM Lomayani S. Laizer 
> wrote:
>
> Hello Lee,
> Yes check can be firewall. Make sure port used by ceph are open
>
> --
> Lomayani
>
> On Fri, Oct 7, 2016 at 6:44 AM, Jaemyoun Lee 
> wrote:
>
> Dear Laizer,
>
> I did deploy the configuration and the key by '$ ceph-deploy admin
> client-node' on admin-node.
> jae@client-node$ ls /etc/ceph
> ceph.client.admin.keyring  ceph.conf  rbdmap  tmpoWLFTb
>
>
>
>
> On Fri, Oct 7, 2016 at 12:33 PM Lomayani S. Laizer 
> wrote:
>
> Hello Lee,
> Make sure you have copied configuration and key for authentication
> (ceph.client.admin.keyring) to client node. Looks it is an authentication
> issue due to missing client key
>
>
> --
> Lomayani
>
> On Fri, Oct 7, 2016 at 5:25 AM, Jaemyoun Lee 
> wrote:
>
> Hi,
> I would like to create a rbd on client-node.
>
> I created a cluster on admin-node successfully.
> jae@admin-node$ ceph health
> HEALTH_OK
>
> A rbd was created on admin-node successfully.
> jae@admin-node$ rbd created foo --size 1024
>
>
> However, when I created a rbd on client-node, the error was occurred:
> jae@client-node$ rbd create foo --size 1024
> 2016-10-06 05:15:35.773242 7f3a52f367c0  0 monclient(hunting):
> authenticate timed out after 300
> 2016-10-06 05:15:35.775497 7f3a52f367c0  0 librados: client.admin
> authentication error (110) Connection timed out
>
> I guess the firewall is right because I can connect from client-node to
> admin-node by ssh.
>
> May you talk me what is wrong?
>
> I referred the following site:
> http://docs.ceph.com/docs/master/start/quick-rbd/
>
> Best regards,
> Jae
>
> --
>
>
> 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된
> 정보가 포함돼 있을 수 있습니다.
>
> 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공개,
> 복사, 전송, 배포해서는 안 됩니다.
>
> 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다.
>
>
> This e-mail is intended only for the named recipient.
>
> Dissemination, distribution, forwarding, or copying of this e-mail by
> anyone other than the intended recipient is prohibited.
>
> If you have received it in error, please notify the sender by e-mail and
> completely delete it. Thank you for your cooperation.
> --
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
> --
>
>
> 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된
> 정보가 포함돼 있을 수 있습니다.
>
> 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공개,
> 복사, 전송, 배포해서는 안 됩니다.
>
> 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다.
>
>
> This e-mail is intended only for the named recipient.
>
> Dissemination, distribution, forwarding, or copying of this e-mail by
> anyone other than the intended recipient is prohibited.
>
> If you have received it in error, please notify the sender by e-mail and
> completely delete it. Thank you for your cooperation.
> --
>
>
>
> --
>
>
> 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된
> 정보가 포함돼 있을 수 있습니다.
>
> 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공개,
> 복사, 전송, 배포해서는 안 됩니다.
>
> 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다.
>
>
> This e-mail is intended only for the named recipient.
>
> Dissemination, distribution, forwarding, or copying of this e-mail by
> anyone other than the intended recipient is prohibited.
>
> If you have received it in error, please notify the sender by e-mail and
> completely delete it. Thank you for your cooperation.
> --
>
>
>
> ___
> ceph-users mailing 
> listceph-us...@lists.ceph.comhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>

---
위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된 정보가 
포함돼 있을 수 있습니다. 
귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를

Re: [ceph-users] creating a rbd

2016-10-06 Thread Daleep Singh Bais

Hi,

Ceph uses other ports also for other daemons like OSD and MDS. Please
refer to

http://docs.ceph.com/docs/jewel/rados/configuration/network-config-ref/

This might help you to resolve the issue.

Thanks.

Daleep Singh Bais

On 10/07/2016 10:14 AM, Jaemyoun Lee wrote:
> Dear Laizer,
>
> Oh, I got it. the rbd was created successfully after I disabled the
> firewall.
> However, when the firewall is enabled and the 6789 port is allowed,
> the authentication error is occurred.
>
> Is there any other port?
>
> On Fri, Oct 7, 2016 at 1:23 PM Lomayani S. Laizer  > wrote:
>
> Hello Lee,
> Yes check can be firewall. Make sure port used by ceph are open
>
> --
> Lomayani
>
> On Fri, Oct 7, 2016 at 6:44 AM, Jaemyoun Lee
> > wrote:
>
> Dear Laizer,
>
> I did deploy the configuration and the key by '$ ceph-deploy
> admin client-node' on admin-node.
> jae@client-node$ ls /etc/ceph
> ceph.client.admin.keyring  ceph.conf  rbdmap  tmpoWLFTb
>
>
>
>
> On Fri, Oct 7, 2016 at 12:33 PM Lomayani S. Laizer
> > wrote:
>
> Hello Lee,
> Make sure you have copied configuration and key for
> authentication (ceph.client.admin.keyring) to client node.
> Looks it is an authentication issue due to missing client key
>
>
> --
> Lomayani
>
> On Fri, Oct 7, 2016 at 5:25 AM, Jaemyoun Lee
> >
> wrote:
>
> Hi, 
> I would like to create a rbd on client-node.
>
> I created a cluster on admin-node successfully.
> jae@admin-node$ ceph health
> HEALTH_OK
>
> A rbd was created on admin-node successfully.
> jae@admin-node$ rbd created foo --size 1024
>
>
> However, when I created a rbd on client-node, the
> error was occurred:
> jae@client-node$ rbd create foo --size 1024
> 2016-10-06 05:15:35.773242 7f3a52f367c0  0
> monclient(hunting): authenticate timed out after 300
> 2016-10-06 05:15:35.775497 7f3a52f367c0  0
> librados: client.admin authentication error (110)
> Connection timed out
>
> I guess the firewall is right because I can connect
> from client-node to admin-node by ssh.
>
> May you talk me what is wrong?
>
> I referred the following
> site: http://docs.ceph.com/docs/master/start/quick-rbd/
>
> Best regards,
> Jae
>
> 
> 
>
>
> 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되
> 는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타
> 사유로 공개가 금지된 정보가 포함돼 있을 수 있습니다.
>
> 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에
> 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거
> 나 제3자에게 공개, 복사, 전송, 배포해서는 안 됩니다.
>
> 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락
> 해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해
> 주셔서 감사합니다.
>
>
> This e-mail is intended only for the named recipient.
>
> Dissemination, distribution, forwarding, or copying of
> this e-mail by anyone other than the intended
> recipient is prohibited.
>
> If you have received it in error, please notify the
> sender by e-mail and completely delete it. Thank you
> for your cooperation.
>
> 
> 
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
> 
> 
>
>
> 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으
> 로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금
> 지된 정보가 포함돼 있을 수 있습니다.
>
> 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된
> 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공
> 개, 복사, 전송, 배포해서는 안 됩니다.
>
> 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시
> 고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다.
>
>
> This e-mail is intended only for the named recipient.
>
> Dissemination, distribution, forwarding, or copying of this
> e-mail by anyone other than the intended recipient is prohibited.
>
> If you have received it in error, please notify the sender by
>

[ceph-users] Hammer OSD memory usage very high

2016-10-06 Thread David Burns

Hello all,

We have a small 160TB Ceph cluster used only as a test s3 storage repository 
for media content.

Problem
Since upgrading from Firefly to Hammer we are experiencing very high OSD memory 
use of 2-3 GB per TB of OSD storage - typical OSD memory 6-10GB.
We have had to increase swap space to bring the cluster to a basic functional 
state. Clearly this will significantly impact system performance and precludes 
starting all OSDs simultaneously.

Hardware
4 x storage nodes with 16 OSDs/node. OSD nodes are reasonable spec SMC storage 
servers with dual Xeon CPUs. Storage is 16 x 3TB SAS disks in each node.
Installed RAM is 72GB (2 nodes) & 80GB (2 nodes). (We note that the installed 
RAM is at least 50% higher than the Ceph recommended 1 GB RAM per TB of 
storage.)

Software
OSD node OS is CentOS 6.8 (with updates). One node has been updated to CentOS 
7.2 - no change in memory usage was observed.

"ceph -v" -> ceph version 0.94.9 (fe6d859066244b97b24f09d46552afc2071e6f90)
(all Ceph packages downloaded from download.ceph.com)

The cluster has achieved status HEALTH_OK so we don’t believe this relates to 
increased memory due to recovery.

History
Emperor 0.72.2 -> Firefly 0.80.10 -> Hammer 0.94.6 -> Hammer 0.94.7 -> Hammer 
0.94.9

OSD per process memory is observed to increase substantially during load_pgs 
phase.

Use of "ceph tell 'osd.*' heap release” has minimal effect - there is no 
substantial memory in the heap or cache freelists.

More information can be found in bug #17228 (link 
http://tracker.ceph.com/issues/17228)

Any feedback or guidance to further understanding the high memory usage would 
be welcomed.

Thanks

David


-- 
FetchTV Pty Ltd, Level 5, 61 Lavender Street, Milsons Point, NSW 2061



This email is sent by FetchTV Pty Ltd (ABN 36 130 669 500). The contents of 
this communication may be 
confidential, legally privileged and/or copyright material. If you are not 
the intended recipient, any use, 
disclosure or copying of this communication is expressed prohibited. If you 
have received this email in error, 
please notify the sender and delete it immediately.


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] creating a rbd

2016-10-06 Thread Jaemyoun Lee

Dear Laizer,

Oh, I got it. the rbd was created successfully after I disabled the
firewall.
However, when the firewall is enabled and the 6789 port is allowed, the
authentication error is occurred.

Is there any other port?

On Fri, Oct 7, 2016 at 1:23 PM Lomayani S. Laizer 
wrote:

> Hello Lee,
> Yes check can be firewall. Make sure port used by ceph are open
>
> --
> Lomayani
>
> On Fri, Oct 7, 2016 at 6:44 AM, Jaemyoun Lee 
> wrote:
>
> Dear Laizer,
>
> I did deploy the configuration and the key by '$ ceph-deploy admin
> client-node' on admin-node.
> jae@client-node$ ls /etc/ceph
> ceph.client.admin.keyring  ceph.conf  rbdmap  tmpoWLFTb
>
>
>
>
> On Fri, Oct 7, 2016 at 12:33 PM Lomayani S. Laizer 
> wrote:
>
> Hello Lee,
> Make sure you have copied configuration and key for authentication
> (ceph.client.admin.keyring) to client node. Looks it is an authentication
> issue due to missing client key
>
>
> --
> Lomayani
>
> On Fri, Oct 7, 2016 at 5:25 AM, Jaemyoun Lee 
> wrote:
>
> Hi,
> I would like to create a rbd on client-node.
>
> I created a cluster on admin-node successfully.
> jae@admin-node$ ceph health
> HEALTH_OK
>
> A rbd was created on admin-node successfully.
> jae@admin-node$ rbd created foo --size 1024
>
>
> However, when I created a rbd on client-node, the error was occurred:
> jae@client-node$ rbd create foo --size 1024
> 2016-10-06 05:15:35.773242 7f3a52f367c0  0 monclient(hunting):
> authenticate timed out after 300
> 2016-10-06 05:15:35.775497 7f3a52f367c0  0 librados: client.admin
> authentication error (110) Connection timed out
>
> I guess the firewall is right because I can connect from client-node to
> admin-node by ssh.
>
> May you talk me what is wrong?
>
> I referred the following site:
> http://docs.ceph.com/docs/master/start/quick-rbd/
>
> Best regards,
> Jae
>
> --
>
>
> 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된
> 정보가 포함돼 있을 수 있습니다.
>
> 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공개,
> 복사, 전송, 배포해서는 안 됩니다.
>
> 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다.
>
>
> This e-mail is intended only for the named recipient.
>
> Dissemination, distribution, forwarding, or copying of this e-mail by
> anyone other than the intended recipient is prohibited.
>
> If you have received it in error, please notify the sender by e-mail and
> completely delete it. Thank you for your cooperation.
> --
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
> --
>
>
> 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된
> 정보가 포함돼 있을 수 있습니다.
>
> 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공개,
> 복사, 전송, 배포해서는 안 됩니다.
>
> 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다.
>
>
> This e-mail is intended only for the named recipient.
>
> Dissemination, distribution, forwarding, or copying of this e-mail by
> anyone other than the intended recipient is prohibited.
>
> If you have received it in error, please notify the sender by e-mail and
> completely delete it. Thank you for your cooperation.
> --
>
>
>

---
위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된 정보가 
포함돼 있을 수 있습니다. 
귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공개, 복사, 
전송, 배포해서는 안 됩니다. 
본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다.

This e-mail is intended only for the named recipient. 
Dissemination, distribution, forwarding, or copying of this e-mail by anyone 
other than the intended recipient is prohibited. 
If you have received it in error, please notify the sender by e-mail and 
completely delete it. Thank you for your cooperation.
---
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] creating a rbd

2016-10-06 Thread Lomayani S. Laizer

Hello Lee,
Yes check can be firewall. Make sure port used by ceph are open

--
Lomayani

On Fri, Oct 7, 2016 at 6:44 AM, Jaemyoun Lee  wrote:

> Dear Laizer,
>
> I did deploy the configuration and the key by '$ ceph-deploy admin
> client-node' on admin-node.
> jae@client-node$ ls /etc/ceph
> ceph.client.admin.keyring  ceph.conf  rbdmap  tmpoWLFTb
>
>
>
>
> On Fri, Oct 7, 2016 at 12:33 PM Lomayani S. Laizer 
> wrote:
>
> Hello Lee,
> Make sure you have copied configuration and key for authentication
> (ceph.client.admin.keyring) to client node. Looks it is an authentication
> issue due to missing client key
>
>
> --
> Lomayani
>
> On Fri, Oct 7, 2016 at 5:25 AM, Jaemyoun Lee 
> wrote:
>
> Hi,
> I would like to create a rbd on client-node.
>
> I created a cluster on admin-node successfully.
> jae@admin-node$ ceph health
> HEALTH_OK
>
> A rbd was created on admin-node successfully.
> jae@admin-node$ rbd created foo --size 1024
>
>
> However, when I created a rbd on client-node, the error was occurred:
> jae@client-node$ rbd create foo --size 1024
> 2016-10-06 05:15:35.773242 7f3a52f367c0  0 monclient(hunting):
> authenticate timed out after 300
> 2016-10-06 05:15:35.775497 7f3a52f367c0  0 librados: client.admin
> authentication error (110) Connection timed out
>
> I guess the firewall is right because I can connect from client-node to
> admin-node by ssh.
>
> May you talk me what is wrong?
>
> I referred the following site: http://docs.ceph.com/
> docs/master/start/quick-rbd/
>
> Best regards,
> Jae
>
> --
>
>
> 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된
> 정보가 포함돼 있을 수 있습니다.
>
> 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공개,
> 복사, 전송, 배포해서는 안 됩니다.
>
> 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다.
>
>
> This e-mail is intended only for the named recipient.
>
> Dissemination, distribution, forwarding, or copying of this e-mail by
> anyone other than the intended recipient is prohibited.
>
> If you have received it in error, please notify the sender by e-mail and
> completely delete it. Thank you for your cooperation.
> --
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
> --
>
>
> 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된
> 정보가 포함돼 있을 수 있습니다.
>
> 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공개,
> 복사, 전송, 배포해서는 안 됩니다.
>
> 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다.
>
>
> This e-mail is intended only for the named recipient.
>
> Dissemination, distribution, forwarding, or copying of this e-mail by
> anyone other than the intended recipient is prohibited.
>
> If you have received it in error, please notify the sender by e-mail and
> completely delete it. Thank you for your cooperation.
> --
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] creating a rbd

2016-10-06 Thread Jaemyoun Lee

Dear Laizer,

I did deploy the configuration and the key by '$ ceph-deploy admin
client-node' on admin-node.
jae@client-node$ ls /etc/ceph
ceph.client.admin.keyring ceph.conf rbdmap tmpoWLFTb

On Fri, Oct 7, 2016 at 12:33 PM Lomayani S. Laizer
wrote:

Hello Lee,
Make sure you have copied configuration and key for authentication
(ceph.client.admin.keyring) to client node. Looks it is an authentication
issue due to missing client key

--
Lomayani

On Fri, Oct 7, 2016 at 5:25 AM, Jaemyoun Lee wrote:

Hi,
I would like to create a rbd on client-node.

I created a cluster on admin-node successfully.
jae@admin-node$ ceph health
HEALTH_OK

A rbd was created on admin-node successfully.
jae@admin-node$ rbd created foo --size 1024

However, when I created a rbd on client-node, the error was occurred:
jae@client-node$ rbd create foo --size 1024
2016-10-06 05:15:35.773242 7f3a52f367c0 0 monclient(hunting):
authenticate timed out after 300
2016-10-06 05:15:35.775497 7f3a52f367c0 0 librados: client.admin
authentication error (110) Connection timed out

I guess the firewall is right because I can connect from client-node to
admin-node by ssh.

May you talk me what is wrong?

I referred the following site:
http://docs.ceph.com/docs/master/start/quick-rbd/

Best regards,
Jae

위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된
정보가 포함돼 있을 수 있습니다.

귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공개,
복사, 전송, 배포해서는 안 됩니다.

본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다.

This e-mail is intended only for the named recipient.

Dissemination, distribution, forwarding, or copying of this e-mail by
anyone other than the intended recipient is prohibited.

If you have received it in error, please notify the sender by e-mail and
completely delete it. Thank you for your cooperation.
--

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

---
위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된 정보가
포함돼 있을 수 있습니다.
귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공개, 복사,
전송, 배포해서는 안 됩니다.
본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다.

This e-mail is intended only for the named recipient.
Dissemination, distribution, forwarding, or copying of this e-mail by anyone
other than the intended recipient is prohibited.
If you have received it in error, please notify the sender by e-mail and
completely delete it. Thank you for your cooperation.
---
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] creating a rbd

2016-10-06 Thread Lomayani S. Laizer

Hello Lee,
Make sure you have copied configuration and key for authentication
(ceph.client.admin.keyring) to client node. Looks it is an authentication
issue due to missing client key


--
Lomayani

On Fri, Oct 7, 2016 at 5:25 AM, Jaemyoun Lee  wrote:

> Hi,
> I would like to create a rbd on client-node.
>
> I created a cluster on admin-node successfully.
> jae@admin-node$ ceph health
> HEALTH_OK
>
> A rbd was created on admin-node successfully.
> jae@admin-node$ rbd created foo --size 1024
>
>
> However, when I created a rbd on client-node, the error was occurred:
> jae@client-node$ rbd create foo --size 1024
> 2016-10-06 05:15:35.773242 7f3a52f367c0  0 monclient(hunting):
> authenticate timed out after 300
> 2016-10-06 05:15:35.775497 7f3a52f367c0  0 librados: client.admin
> authentication error (110) Connection timed out
>
> I guess the firewall is right because I can connect from client-node to
> admin-node by ssh.
>
> May you talk me what is wrong?
>
> I referred the following site: http://docs.ceph.com/
> docs/master/start/quick-rbd/
>
> Best regards,
> Jae
>
> --
>
>
> 위 전자우편에 포함된 정보는 지정된 수신인에게만 발송되는 것으로 보안을 유지해야 하는 정보와 법률상 및 기타 사유로 공개가 금지된
> 정보가 포함돼 있을 수 있습니다.
>
> 귀하가 이 전자우편의 지정 수신인이 아니라면 본 메일에 포함된 정보의 전부 또는 일부를 무단으로 보유, 사용하거나 제3자에게 공개,
> 복사, 전송, 배포해서는 안 됩니다.
>
> 본 메일이 잘못 전송되었다면, 전자우편 혹은 전화로 연락해주시고, 메일을 즉시 삭제해 주시기 바랍니다. 협조해 주셔서 감사합니다.
>
>
> This e-mail is intended only for the named recipient.
>
> Dissemination, distribution, forwarding, or copying of this e-mail by
> anyone other than the intended recipient is prohibited.
>
> If you have received it in error, please notify the sender by e-mail and
> completely delete it. Thank you for your cooperation.
> --
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] creating a rbd

2016-10-06 Thread Jaemyoun Lee

Hi,
I would like to create a rbd on client-node.

I created a cluster on admin-node successfully.
jae@admin-node$ ceph health
HEALTH_OK

A rbd was created on admin-node successfully.
jae@admin-node$ rbd created foo --size 1024

I guess the firewall is right because I can connect from client-node to
admin-node by ssh.

May you talk me what is wrong?

I referred the following site:
http://docs.ceph.com/docs/master/start/quick-rbd/

Best regards,
Jae

This e-mail is intended only for the named recipient.
Dissemination, distribution, forwarding, or copying of this e-mail by anyone
other than the intended recipient is prohibited.
If you have received it in error, please notify the sender by e-mail and
completely delete it. Thank you for your cooperation.
---___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] jewel/CephFS - misc problems (duplicate strays, mismatch between head items and fnode.fragst)

2016-10-06 Thread Kjetil Jørgensen

And - I just saw another recent thread - http://tracker.ceph.com/
issues/17177 - can be an explanation of most/all of the above ?

Next question(s) would then be:

   - How would one deal with duplicate stray(s)
   - How would one deal with mismatch between head items and
   fnode.fragstat, ceph daemon mds.foo scrub_path ?

-KJ

On Thu, Oct 6, 2016 at 5:05 PM, Kjetil Jørgensen 
wrote:

> Hi,
>
> context (i.e. what we're doing): We're migrating (or trying to) migrate
> off of an nfs server onto cephfs, for a workload that's best described as
> "big piles" of hardlinks. Essentially, we have a set of "sources":
> foo/01/
> foo/0b/<0b>
> .. and so on
> bar/02/..
> bar/0c/..
> .. and so on
>
> foo/bar/friends have been "cloned" numerous times to a set of names that
> over the course of weeks end up being recycled again, the clone is
> essentially cp -L foo copy-1-of-foo.
>
> We're doing "incremental" rsyncs of this onto cephfs, so the sense of "the
> original source of the hardlink" will end up moving around, depending on
> the whims of rsync. (if it matters, I found some allusion to "if the
> original file hardlinked is deleted, ...".
>
> For RBD the ceph cluster have mostly been rather well behaved, the
> problems we have had have for the most part been self-inflicted. Before
> introducing the hardlink spectacle to cephfs, the same filesystem were used
> for light-ish read-mostly loads, beint mostly un-eventful. (That being
> said, we did patch it for
>
> Cluster is v10.2.2 (mds v10.2.2+4d15eb12298e007744486e28924a6f0ae071bd06),
> clients are ubuntu's 4.4.0-32 kernel(s), and elrepo v4.4.4.
>
> The problems we're facing:
>
>- Maybe a "non-problem" I have ~6M strays sitting around
>- Slightly more problematic, I have duplicate stray(s) ? See log
>excercepts below. Also; rados -p cephfs_metadata listomapkeys 60X.
>did/does seem to agree with there being duplicate strays (assuming
>60X. is the directory indexes for the stray catalogs), caveat "not
>a perfect snapshot", listomapkeys issued in serial fashion.
>- We stumbled across (http://tracker.ceph.com/issues/17177 - mostly
>here for more context)
>- There's been a couple of instances of invalid backtrace(s), mostly
>solved by either mds:scrub_path or just unlinking the files/directories in
>question and re-rsync-ing.
>- mismatch between head items and fnode.fragstat (See below for more
>of the log excercept), appeared to have been solved by mds:scrub_path
>
>
> Duplicate stray(s), ceph-mds complains (a lot, during rsync):
> 2016-09-30 20:00:21.978314 7ffb653b8700  0 mds.0.cache.dir(603) _fetched
>  badness: got (but i already had) [inode 10003f25eaf [...2,head]
> ~mds0/stray0/10003f25eaf auth v38836572 s=8998 nl=5 n(v0 b8998 1=1+0)
> (iversion lock) 0x561082e6b520] mode 33188 mtime 2016-07-25 03:02:50.00
> 2016-09-30 20:00:21.978336 7ffb653b8700 -1 log_channel(cluster) log [ERR]
> : loaded dup inode 10003f25eaf [2,head] v36792929 at
> ~mds0/stray3/10003f25eaf, but inode 10003f25eaf.head v38836572 already
> exists at ~mds0/stray0/10003f25eaf
>
> I briefly ran ceph-mds with debug_mds=20/20 which didn't yield anything
> immediately useful, beyond slightly-easier-to-follow the control-flow
> of src/mds/CDir.cc without becoming much wiser.
> 2016-09-30 20:43:51.910754 7ffb653b8700 20 mds.0.cache.dir(606) _fetched
> pos 310473 marker 'I' dname '100022e8617 [2,head]
> 2016-09-30 20:43:51.910757 7ffb653b8700 20 mds.0.cache.dir(606) lookup
> (head, '100022e8617')
> 2016-09-30 20:43:51.910759 7ffb653b8700 20 mds.0.cache.dir(606)   miss ->
> (10002a81c10,head)
> 2016-09-30 20:43:51.910762 7ffb653b8700  0 mds.0.cache.dir(606) _fetched
>  badness: got (but i already had) [inode 100022e8617 [...2,head]
> ~mds0/stray9/100022e8617 auth v39303851 s=11470 nl=10 n(v0 b11470 1=1+0)
> (iversion lock) 0x560c013904b8] mode 33188 mtime 2016-07-25 03:38:01.00
> 2016-09-30 20:43:51.910772 7ffb653b8700 -1 log_channel(cluster) log [ERR]
> : loaded dup inode 100022e8617 [2,head] v39284583 at
> ~mds0/stray6/100022e8617, but inode 100022e8617.head v39303851 already
> exists at ~mds0/stray9/100022e8617
>
>
> 2016-09-25 06:23:50.947761 7ffb653b8700  1 mds.0.cache.dir(10003439a33)
> mismatch between head items and fnode.fragstat! printing dentries
> 2016-09-25 06:23:50.947779 7ffb653b8700  1 mds.0.cache.dir(10003439a33)
> get_num_head_items() = 36; fnode.fragstat.nfiles=53
> fnode.fragstat.nsubdirs=0
> 2016-09-25 06:23:50.947782 7ffb653b8700  1 mds.0.cache.dir(10003439a33)
> mismatch between child accounted_rstats and my rstats!
> 2016-09-25 06:23:50.947803 7ffb653b8700  1 mds.0.cache.dir(10003439a33)
> total of child dentrys: n(v0 b19365007 36=36+0)
> 2016-09-25 06:23:50.947806 7ffb653b8700  1 mds.0.cache.dir(10003439a33) my
> rstats:  n(v2 rc2016-08-28 04:48:37.685854 b49447206 53=53+0)
>
> The slightly sad thing is - I suspect all of this is probably from
> something that "happened

[ceph-users] jewel/CephFS - misc problems (duplicate strays, mismatch between head items and fnode.fragst)

2016-10-06 Thread Kjetil Jørgensen

Hi,

context (i.e. what we're doing): We're migrating (or trying to) migrate off
of an nfs server onto cephfs, for a workload that's best described as "big
piles" of hardlinks. Essentially, we have a set of "sources":
foo/01/
foo/0b/<0b>
.. and so on
bar/02/..
bar/0c/..
.. and so on

foo/bar/friends have been "cloned" numerous times to a set of names that
over the course of weeks end up being recycled again, the clone is
essentially cp -L foo copy-1-of-foo.

We're doing "incremental" rsyncs of this onto cephfs, so the sense of "the
original source of the hardlink" will end up moving around, depending on
the whims of rsync. (if it matters, I found some allusion to "if the
original file hardlinked is deleted, ...".

For RBD the ceph cluster have mostly been rather well behaved, the problems
we have had have for the most part been self-inflicted. Before introducing
the hardlink spectacle to cephfs, the same filesystem were used for
light-ish read-mostly loads, beint mostly un-eventful. (That being said, we
did patch it for

Cluster is v10.2.2 (mds v10.2.2+4d15eb12298e007744486e28924a6f0ae071bd06),
clients are ubuntu's 4.4.0-32 kernel(s), and elrepo v4.4.4.

The problems we're facing:

   - Maybe a "non-problem" I have ~6M strays sitting around
   - Slightly more problematic, I have duplicate stray(s) ? See log
   excercepts below. Also; rados -p cephfs_metadata listomapkeys 60X.
   did/does seem to agree with there being duplicate strays (assuming
   60X. is the directory indexes for the stray catalogs), caveat "not
   a perfect snapshot", listomapkeys issued in serial fashion.
   - We stumbled across (http://tracker.ceph.com/issues/17177 - mostly here
   for more context)
   - There's been a couple of instances of invalid backtrace(s), mostly
   solved by either mds:scrub_path or just unlinking the files/directories in
   question and re-rsync-ing.
   - mismatch between head items and fnode.fragstat (See below for more of
   the log excercept), appeared to have been solved by mds:scrub_path


Duplicate stray(s), ceph-mds complains (a lot, during rsync):
2016-09-30 20:00:21.978314 7ffb653b8700  0 mds.0.cache.dir(603) _fetched
 badness: got (but i already had) [inode 10003f25eaf [...2,head]
~mds0/stray0/10003f25eaf auth v38836572 s=8998 nl=5 n(v0 b8998 1=1+0)
(iversion lock) 0x561082e6b520] mode 33188 mtime 2016-07-25 03:02:50.00
2016-09-30 20:00:21.978336 7ffb653b8700 -1 log_channel(cluster) log [ERR] :
loaded dup inode 10003f25eaf [2,head] v36792929 at
~mds0/stray3/10003f25eaf, but inode 10003f25eaf.head v38836572 already
exists at ~mds0/stray0/10003f25eaf

I briefly ran ceph-mds with debug_mds=20/20 which didn't yield anything
immediately useful, beyond slightly-easier-to-follow the control-flow
of src/mds/CDir.cc without becoming much wiser.
2016-09-30 20:43:51.910754 7ffb653b8700 20 mds.0.cache.dir(606) _fetched
pos 310473 marker 'I' dname '100022e8617 [2,head]
2016-09-30 20:43:51.910757 7ffb653b8700 20 mds.0.cache.dir(606) lookup
(head, '100022e8617')
2016-09-30 20:43:51.910759 7ffb653b8700 20 mds.0.cache.dir(606)   miss ->
(10002a81c10,head)
2016-09-30 20:43:51.910762 7ffb653b8700  0 mds.0.cache.dir(606) _fetched
 badness: got (but i already had) [inode 100022e8617 [...2,head]
~mds0/stray9/100022e8617 auth v39303851 s=11470 nl=10 n(v0 b11470 1=1+0)
(iversion lock) 0x560c013904b8] mode 33188 mtime 2016-07-25 03:38:01.00
2016-09-30 20:43:51.910772 7ffb653b8700 -1 log_channel(cluster) log [ERR] :
loaded dup inode 100022e8617 [2,head] v39284583 at
~mds0/stray6/100022e8617, but inode 100022e8617.head v39303851 already
exists at ~mds0/stray9/100022e8617


2016-09-25 06:23:50.947761 7ffb653b8700  1 mds.0.cache.dir(10003439a33)
mismatch between head items and fnode.fragstat! printing dentries
2016-09-25 06:23:50.947779 7ffb653b8700  1 mds.0.cache.dir(10003439a33)
get_num_head_items() = 36; fnode.fragstat.nfiles=53
fnode.fragstat.nsubdirs=0
2016-09-25 06:23:50.947782 7ffb653b8700  1 mds.0.cache.dir(10003439a33)
mismatch between child accounted_rstats and my rstats!
2016-09-25 06:23:50.947803 7ffb653b8700  1 mds.0.cache.dir(10003439a33)
total of child dentrys: n(v0 b19365007 36=36+0)
2016-09-25 06:23:50.947806 7ffb653b8700  1 mds.0.cache.dir(10003439a33) my
rstats:  n(v2 rc2016-08-28 04:48:37.685854 b49447206 53=53+0)

The slightly sad thing is - I suspect all of this is probably from
something that "happened at some time in the past", and running mds with
debugging will make my users very unhappy as writing/formatting all that
log is not exactly cheap. (debug_mds=20/20, quickly ended up with mds
beacon marked as laggy).

Bonus question: In terms of "understanding how cephfs works" is
doc/dev/mds_internals it ? :) Given that making "minimal reproducible
test-cases" so far is turning to be quite elusive from the "top down"
approach, I'm finding myself looking inside the box to try to figure out
how we got where we are.

(And many thanks for ceph-dencoder, it satisfies my

Re: [ceph-users] unable to start radosgw after upgrade from 10.2.2 to 10.2.3

2016-10-06 Thread Andrei Mikhailovsky

Hi Graham,

Yeah, I am not sure why no one else is having the same issues. Anyway, had a 
chat on irc and got a link that helped me: 
https://www.mail-archive.com/ceph-users@lists.ceph.com/msg31764.html

I've followed what it said, even though the errors i got were different, but it 
helped me to start the service. I am yet to test if the rgw is functional and 
user clients can connect.

Hope that helps

andrei

- Original Message -
> From: "Graham Allan" 
> To: "ceph-users" 
> Sent: Thursday, 6 October, 2016 20:04:38
> Subject: Re: [ceph-users] unable to start radosgw after upgrade from 10.2.2 
> to 10.2.3

> That's interesting, as I am getting the exact same errors after
> upgrading from Hammer 0.94.9 to Jewel 10.2.3 (on ubuntu 14.04).
> 
> I wondered if it was the issue referred to a few months ago here, but
> I'm not so sure, since the error returned from radosgw-admin commands is
> different:
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-April/009171.html
> 
> I do have one radosgw which is still on 0.94.9 and still functions
> normally - is it possible that this is preventing the config migration
> alluded to in that thread? I'm reluctant to do anything to the
> still-working 0.94.9 gateway until I can get the 10.2.3 gateways working!
> 
> Graham
> 
> On 10/05/2016 04:23 PM, Andrei Mikhailovsky wrote:
>> Hello everyone,
>>
>> I've just updated my ceph to version 10.2.3 from 10.2.2 and I am no
>> longer able to start the radosgw service. When executing I get the
>> following error:
>>
>> 2016-10-05 22:14:10.735883 7f1852d26a00  0 ceph version 10.2.3
>> (ecc23778eb545d8dd55e2e4735b53cc93f92e65b), process radosgw, pid 2711
>> 2016-10-05 22:14:10.765648 7f1852d26a00  0 pidfile_write: ignore empty
>> --pid-file
>> 2016-10-05 22:14:11.287772 7f1852d26a00  0 zonegroup default missing
>> zone for master_zone=
>> 2016-10-05 22:14:11.294141 7f1852d26a00 -1 Couldn't init storage
>> provider (RADOS)
>>
>>
>>
>> I had no issues starting rados on 10.2.2 and all versions prior to that.
>>
>> I am running ceph 10.2.3 on Ubuntu 16.04 LTS servers.
>>
>> Could someone please help me with fixing the problem?
>>
>> Thanks
>>
>> Andrei
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> 
> --
> Graham Allan
> Minnesota Supercomputing Institute - g...@umn.edu
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Migrate pre-Jewel RGW data to Jewel realm/zonegroup/zone?

2016-10-06 Thread Richard Chan

Thanks, that's just what I needed.

For others: I found some information in the Red Hat Ceph Storage 2
documentation. This includes the command "radosgw-admin rename"
which I was unaware of for single-site to multi-site.

(It doesn't seem very encouraging about Hammer to Jewel multisite
transitions:
some vague statement "Customers wishing to migrate from a 1.3 multi-site
 system, formerly called "federated", contact customer support for advice
on recommended steps so Red Hat Support can look at the configuration,
environment, and data first." - Yikes.

https://access.redhat.com/documentation/en/red-hat-ceph-storage/2/paged/object-gateway-guide-for-red-hat-enterprise-linux/chapter-8-multi-site

On Thu, Oct 6, 2016 at 11:23 PM, Yehuda Sadeh-Weinraub 
wrote:

> Generally you need to create a new realm, and add the 'default'
> zonegroup into it. I think you can achieve this via the 'radosgw-admin
> zonegroup modify' command.
> The zonegroup and zone can be renamed (their id will still be
> 'default', but you can change their names).
>
> Yehuda
>
> On Thu, Oct 6, 2016 at 3:07 AM, Richard Chan
>  wrote:
> > Upgraded from Hammer 0.94.9 to Jewel 10.2.3 while all RGW data survived
> in a
> > realm-less, setup (no realm, "default" zonegroup, "default" zone).
> >
> > Is it possible to "move" this into a single realm/zonegroup/zone in
> > preparation for multisite (i.e before add the 2nd zone).
> >
> > I don't need more than one realm, but would like to
> >
> > 1. create a single realm"default"
> > 2. move zonegroup "default" as "us", say
> > 3. move zone to zonegroup as "us-east"
> >
> > This must preserve all existing data. After which I would like to add a
> 2nd
> > zone (in a different ceph cluster) "us-west".
> >
> >
> > # radosgw-admin realm list
> > {
> > "default_info": "",
> > "realms": []
> > }
> >
> > # radosgw-admin zonegroup list
> > read_default_id : -2
> > {
> > "default_info": "",
> > "zonegroups": [
> > "default"
> > ]
> > }
> >
> >
> > # radosgw-admin zone list
> > {
> > "default_info": "",
> > "zones": [
> > "default"
> > ]
> > }
> >
> >
> >
> >
> >
> >
> > --
> > Richard Chan
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>



-- 
Richard Chan
Chief Architect

TreeBox Solutions Pte Ltd
1 Commonwealth Lane #03-01
Singapore 149544
Tel: 6570 3725
http://www.treeboxsolutions.com

Co.Reg.No. 201100585R
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] unable to start radosgw after upgrade from 10.2.2 to 10.2.3

2016-10-06 Thread Graham Allan

That's interesting, as I am getting the exact same errors after 
upgrading from Hammer 0.94.9 to Jewel 10.2.3 (on ubuntu 14.04).


I wondered if it was the issue referred to a few months ago here, but 
I'm not so sure, since the error returned from radosgw-admin commands is 
different:

http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-April/009171.html

I do have one radosgw which is still on 0.94.9 and still functions 
normally - is it possible that this is preventing the config migration 
alluded to in that thread? I'm reluctant to do anything to the 
still-working 0.94.9 gateway until I can get the 10.2.3 gateways working!


Graham

On 10/05/2016 04:23 PM, Andrei Mikhailovsky wrote:

Hello everyone,

I've just updated my ceph to version 10.2.3 from 10.2.2 and I am no
longer able to start the radosgw service. When executing I get the
following error:

2016-10-05 22:14:10.735883 7f1852d26a00  0 ceph version 10.2.3
(ecc23778eb545d8dd55e2e4735b53cc93f92e65b), process radosgw, pid 2711
2016-10-05 22:14:10.765648 7f1852d26a00  0 pidfile_write: ignore empty
--pid-file
2016-10-05 22:14:11.287772 7f1852d26a00  0 zonegroup default missing
zone for master_zone=
2016-10-05 22:14:11.294141 7f1852d26a00 -1 Couldn't init storage
provider (RADOS)



I had no issues starting rados on 10.2.2 and all versions prior to that.

I am running ceph 10.2.3 on Ubuntu 16.04 LTS servers.

Could someone please help me with fixing the problem?

Thanks

Andrei


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] upgrade from v0.94.6 or lower and 'failed to encode map X with expected crc'

2016-10-06 Thread Stillwell, Bryan J

Thanks Kefu!

Downgrading the mons to 0.94.6 got us out of this situation.  I appreciate
you tracking this down!

Bryan

On 10/4/16, 1:18 AM, "ceph-users on behalf of kefu chai"
 wrote:

>hi ceph users,
>
>If user upgrades the cluster from a prior release to v0.94.7 or up by
>following the steps:
>
>1. upgrade the monitors first,
>2. and then the OSDs.
>
>It is expected that the cluster log will be flooded with messages like:
>
>2016-07-12 08:42:42.1234567 osd.1234 [WRN] failed to encode map e4321
>with expected crc
>
>Because we changed[1] the encoding of OSDMap in v0.94.7. And the
>monitors start sending the incremental OSDMaps with the new encoding
>to the OSDs once the quorum members are all at the new version. But
>the OSDs at the old version still re-encode the osdmaps with the old
>encoding, then compare the resulting CRC with the one carried by the
>received incremental maps. And, they don't match! So the OSDs will ask
>the monitors for the full map in this case.
>
>For a large Ceph cluster, there are several consequences of the CRC
>mismatch:
>1. monitor being flooded by this clog
>2. monitor burdened by the sending the fullmaps.
>3. the network saturated by the osdmap messages carrying the requested
>fullmaps
>3. slow requests observed if the updated osdmaps are delayed by the
>saturated network.
>
>as reported[2,3,4,5] by our users.
>
>The interim solution for those who are stuck in the middle of an upgrade
>is:
>
>1. revert all the monitors back to the previous version,
>2. upgrade the OSDs to the version you want to upgrade.
>3. upgrade the monitors to the version you want to upgrade.
>
>And for users who plan to upgrade from a version prior to v0.94.7 to
>v0.94.7 or up, please
>1. upgrade the OSDs to the version you want to upgrade
>2. upgrade the monitors to the version you want to upgrade.
>
>For users preferring upgrading from a version prior to v0.94.7 to
>jewel, it is suggested to upgrade to the latest hammer first by
>following the steps above, if the scale of your cluster is relatively
>large.
>
>And in the short term, we are preparing a fix[6] for hammer, so the
>monitors will send osdmap encoded with lower version encoding.
>
>In the long term, we won't use the new release feature bit in the
>cluster unless allowed explicitly[7].
>
>
>@ceph developers,
>
>so if we want to bump up the encoding version of OSDMap or its
>(sub)fields, I think it would be desirable to match the encoder with
>the new major release feature bit. For instance, if a new field named
>"foo" is added to `pg_pool_t` in kraken, and `map
>pools` is in turn a field of `OSDMap`, then we need to be careful when
>updating `pg_pool_t::encode()`, like
>
>void pg_pool_t::encode(bufferlist& bl, uint64_t features) const {
>  // ...
>  if ((features & CEPH_FEATURE_SERVER_KRAKEN) == 0) {
>// encode in the jewel way
>return;
>  }
>  // encode in the kraken way
>}
>
>Because,
>
>- it would be difficult for the monitor to send understandable osdmaps
>for all osds.
>- we disable/enable the new encoder by excluding/including the major
>release feature bit in [7].
>
>--
>[1] sha1 039240418060c9a49298dacc0478772334526dce
>[2] https://www.mail-archive.com/ceph-users@lists.ceph.com/msg30783.html
>[3] http://www.spinics.net/lists/ceph-users/msg28296.html
>[4] http://ceph-users.ceph.narkive.com/rPGrATpE/v0-94-7-hammer-released
>[5] 
>http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-September/013189.
>html
>[6] http://tracker.ceph.com/issues/17386
>[7] https://github.com/ceph/ceph/pull/11284
>
>-- 
>Regards
>Kefu Chai
>___
>ceph-users mailing list
>ceph-users@lists.ceph.com
>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Problem copying a file with ceph-fuse

2016-10-06 Thread Gregory Farnum

On Thu, Oct 6, 2016 at 8:28 AM, Dennis Kramer (DBS)  wrote:
> Hi all,
>
> I have an issue that when I copy a specific file with ceph-fuse on
> cephfs (within the same directory) it stalls after a couple of GB of
> data. Nothing happens. No error, it just "hangs".
>
> When I copy the same file with the cephfs kernel client it works without
> issues.
>
> I'm running Jewel 10.2.2 on the cluster. I have tried the ceph-fuse
> client v10.2.2 and v10.2.3.
>
> Any tips on how to debug this issue is more then welcome.

http://docs.ceph.com/docs/master/cephfs/troubleshooting/
In particular the ceph-fuse and MDS portions. ;) Also check out the
objecter's ops in flight on ceph-fuse.
-Greg

>
> PS: it is possible that this specific file has some bitrot, maybe even
> corrupted. But I would like to have an explanation why it works with the
> cephfs kernel client and not ceph-fuse.
>
> With kind regards,
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Appending to an erasure coded pool

2016-10-06 Thread Gregory Farnum

On Thu, Oct 6, 2016 at 4:08 AM, James Norman  wrote:
> Hi there,
>
> I am developing a web application that supports browsing, uploading,
> downloading, moving files in Ceph Rados pool. Internally to write objects we
> use rados_append, as it's often too memory intensive for us to have the full
> file in memory to do a rados_write_full.
>
> We do not control our customer's Ceph installations, such as whether they
> use replicated pools, EC pools etc. We've found that when dealing with a EC
> pool, our rados_append calls return error code 95 and message "Operation not
> supported".
>
> I've had several discussions with members in the IRC chatroom regarding
> this, and the general consensus I've got is:
> 1) Use write alignment.
> 2) Put a replicated pool in front of the EC pool
> 3) EC pools have a limited feature set
>
> Regarding point 1), are there any actual code example for how you would
> handle this in the context of rados_append? I have struggled to find even
> one. This seems to me something that should be handled by either the API
> libraries, or Ceph itself, not the client trying to write some data.

librados requires a fair bit of knowledge from the user applications,
yes. One thing you mention that sounds concerning is that you can't
hold the objects in-memory — RADOS is not comfortable with very large
objects and you'll find that things like backfill might not perform as
you expect. (At this point everything will *probably* function, but it
may be so slow as to make no difference to you when it hits that
situation.) Certainly if your objects do not all fit neatly into
buckets of a particular size and you have some that are very large,
you will have a very not-uniform balance.

But, if you want to learn about EC pools there is some documentation
at http://docs.ceph.com/docs/master/dev/osd_internals/erasure_coding/
(or in ceph.git/doc/dev/osd_internals/erasure_coding) from when they
were being created.

>
> Regarding point 2) This seems to be a workaround, and generally not
> something we want to recommend to our customers. Is it detrimental to us an
> EC pool without a replicated pool? What are the performance costs of doing
> so?

Yeah, don't do that. Cache pools are really tricky to use properly and
turned out not to perform very well.

>
> Regarding point 3) Can you point me towards resources that describe what
> features / abilities you lose by adopting an EC pool?

Same as above links, apparently. But really, you can read from and
append to them. There are no object classes, no arbitrary overwrites,
no omaps.
-Greg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Migrate pre-Jewel RGW data to Jewel realm/zonegroup/zone?

2016-10-06 Thread Yehuda Sadeh-Weinraub

Generally you need to create a new realm, and add the 'default'
zonegroup into it. I think you can achieve this via the 'radosgw-admin
zonegroup modify' command.
The zonegroup and zone can be renamed (their id will still be
'default', but you can change their names).

Yehuda

On Thu, Oct 6, 2016 at 3:07 AM, Richard Chan
 wrote:
> Upgraded from Hammer 0.94.9 to Jewel 10.2.3 while all RGW data survived in a
> realm-less, setup (no realm, "default" zonegroup, "default" zone).
>
> Is it possible to "move" this into a single realm/zonegroup/zone in
> preparation for multisite (i.e before add the 2nd zone).
>
> I don't need more than one realm, but would like to
>
> 1. create a single realm"default"
> 2. move zonegroup "default" as "us", say
> 3. move zone to zonegroup as "us-east"
>
> This must preserve all existing data. After which I would like to add a 2nd
> zone (in a different ceph cluster) "us-west".
>
>
> # radosgw-admin realm list
> {
> "default_info": "",
> "realms": []
> }
>
> # radosgw-admin zonegroup list
> read_default_id : -2
> {
> "default_info": "",
> "zonegroups": [
> "default"
> ]
> }
>
>
> # radosgw-admin zone list
> {
> "default_info": "",
> "zones": [
> "default"
> ]
> }
>
>
>
>
>
>
> --
> Richard Chan
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Problem copying a file with ceph-fuse

2016-10-06 Thread Dennis Kramer (DBS)

Thanks for your response. But yes, the netwerk is OK.
But i will double check to be sure.

Then again, If I copy other (big) files from the same client everything
works without any issues. The problem is isolated to a specific file.
With a mis-configured network I would see this kind of issues constantly.

With kind regards,

On 10/06/2016 04:39 PM, Oliver Dzombic wrote:
> Hi,
> 
> are you sure, that the network is OK ?
> 
> Stuff like this can happen if somewhere in between the MTU size is
> different.
> 
> So make sure, that all public ceph nic's have the same MTU as the
> client, and also every switchport has the same MTU in between.
> 

-- 
Kramer M.D.
Infrastructure Engineer

Nederlands Forensisch Instituut
Digitale Technologie & Biometrie
Laan van Ypenburg 6 | 2497 GB | Den Haag
Postbus 24044 | 2490 AA | Den Haag

T 070 888 64 30
M 06 29 62 12 02
d.kra...@nfi.minvenj.nl / den...@holmes.nl
PGP publickey: http://www.holmes.nl/dennis.asc
www.forensischinstituut.nl

Nederlands Forensisch Instituut. In feiten het beste.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Problem copying a file with ceph-fuse

2016-10-06 Thread Oliver Dzombic

Hi,

are you sure, that the network is OK ?

Stuff like this can happen if somewhere in between the MTU size is
different.

So make sure, that all public ceph nic's have the same MTU as the
client, and also every switchport has the same MTU in between.

-- 
Mit freundlichen Gruessen / Best regards

Oliver Dzombic
IP-Interactive

mailto:i...@ip-interactive.de

Anschrift:

IP Interactive UG ( haftungsbeschraenkt )
Zum Sonnenberg 1-3
63571 Gelnhausen

HRB 93402 beim Amtsgericht Hanau
Geschäftsführung: Oliver Dzombic

Steuer Nr.: 35 236 3622 1
UST ID: DE274086107


Am 06.10.2016 um 16:28 schrieb Dennis Kramer (DBS):
> Hi all,
> 
> I have an issue that when I copy a specific file with ceph-fuse on
> cephfs (within the same directory) it stalls after a couple of GB of
> data. Nothing happens. No error, it just "hangs".
> 
> When I copy the same file with the cephfs kernel client it works without
> issues.
> 
> I'm running Jewel 10.2.2 on the cluster. I have tried the ceph-fuse
> client v10.2.2 and v10.2.3.
> 
> Any tips on how to debug this issue is more then welcome.
> 
> PS: it is possible that this specific file has some bitrot, maybe even
> corrupted. But I would like to have an explanation why it works with the
> cephfs kernel client and not ceph-fuse.
> 
> With kind regards,
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Problem copying a file with ceph-fuse

2016-10-06 Thread Dennis Kramer (DBS)

Hi all,

I have an issue that when I copy a specific file with ceph-fuse on
cephfs (within the same directory) it stalls after a couple of GB of
data. Nothing happens. No error, it just "hangs".

When I copy the same file with the cephfs kernel client it works without
issues.

I'm running Jewel 10.2.2 on the cluster. I have tried the ceph-fuse
client v10.2.2 and v10.2.3.

Any tips on how to debug this issue is more then welcome.

PS: it is possible that this specific file has some bitrot, maybe even
corrupted. But I would like to have an explanation why it works with the
cephfs kernel client and not ceph-fuse.

With kind regards,
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph + VMWare

2016-10-06 Thread Alex Gorbachev

On Wed, Oct 5, 2016 at 2:32 PM, Patrick McGarry  wrote:
> Hey guys,
>
> Starting to buckle down a bit in looking at how we can better set up
> Ceph for VMWare integration, but I need a little info/help from you
> folks.
>
> If you currently are using Ceph+VMWare, or are exploring the option,
> I'd like some simple info from you:
>
> 1) Company
> 2) Current deployment size
> 3) Expected deployment growth
> 4) Integration method (or desired method) ex: iscsi, native, etc
>
> Just casting the net so we know who is interested and might want to
> help us shape and/or test things in the future if we can make it
> better. Thanks.
>

Hi Patrick,

We have Storcium certified with VMWare, and we use it ourselves:

Ceph Hammer latest

SCST redundant Pacemaker based delivery front ends - our agents are
published on github

EnhanceIO for read caching at delivery layer

NFS v3, and iSCSI and FC delivery

Our deployment size we use ourselves is 700 TB raw.

Challenges are as others described, but HA and multi host access works
fine courtesy of SCST.  Write amplification is a challenge on spinning
disks.

Happy to share more.

Alex

>
> --
>
> Best Regards,
>
> Patrick McGarry
> Director Ceph Community || Red Hat
> http://ceph.com  ||  http://community.redhat.com
> @scuttlemonkey || @ceph
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Can't activate OSD

2016-10-06 Thread Stas Starikevich

Hi,

Faced with the similar problems on the CentOS7 - looks like condition
race with parted.
Update to 3.2 solve my problem (from 3.1 from the CentOS7 base):

rpm -Uhv 
ftp://195.220.108.108/linux/fedora/linux/updates/22/x86_64/p/parted-3.2-16.fc22.x86_64.rpm

Stas


On Mon, Oct 3, 2016 at 6:39 PM, Tracy Reed  wrote:
> Oops, I said CentOS 5 (old habit, ran it for years!). I meant CentOS 7. And 
> I'm
> running the following Ceph package versions from the ceph repo:
>
> root@ceph02 ~]# rpm -qa |grep -i ceph
> libcephfs1-10.2.3-0.el7.x86_64
> ceph-common-10.2.3-0.el7.x86_64
> ceph-mon-10.2.3-0.el7.x86_64
> ceph-release-1-1.el7.noarch
> python-cephfs-10.2.3-0.el7.x86_64
> ceph-selinux-10.2.3-0.el7.x86_64
> ceph-osd-10.2.3-0.el7.x86_64
> ceph-mds-10.2.3-0.el7.x86_64
> ceph-radosgw-10.2.3-0.el7.x86_64
> ceph-base-10.2.3-0.el7.x86_64
> ceph-10.2.3-0.el7.x86_64
>
> On Mon, Oct 03, 2016 at 03:34:50PM PDT, Tracy Reed spake thusly:
>> Hello all,
>>
>> Over the past few weeks I've been trying to go through the Quick Ceph Deploy 
>> tutorial at:
>>
>> http://docs.ceph.com/docs/jewel/start/quick-ceph-deploy/
>>
>> just trying to get a basic 2 OSD ceph cluster up and running. Everything 
>> seems
>> to go well until I get to the:
>>
>> ceph-deploy osd activate ceph02:/dev/sdc ceph03:/dev/sdc
>>
>> part. It never actually seems to activate the OSD and eventually times out:
>>
>> [ceph02][DEBUG ] connection detected need for sudo
>> [ceph02][DEBUG ] connected to host: ceph02
>> [ceph02][DEBUG ] detect platform information from remote host
>> [ceph02][DEBUG ] detect machine type
>> [ceph02][DEBUG ] find the location of an executable
>> [ceph_deploy.osd][INFO  ] Distro info: CentOS Linux 7.2.1511 Core
>> [ceph_deploy.osd][DEBUG ] activating host ceph02 disk /dev/sdc
>> [ceph_deploy.osd][DEBUG ] will use init type: systemd
>> [ceph02][DEBUG ] find the location of an executable
>> [ceph02][INFO  ] Running command: sudo /usr/sbin/ceph-disk -v activate 
>> --mark-init systemd --mount /dev/sdc
>> [ceph02][WARNIN] main_activate: path = /dev/sdc
>> [ceph02][WARNIN] No data was received after 300 seconds, disconnecting...
>> [ceph02][INFO  ] checking OSD status...
>> [ceph02][DEBUG ] find the location of an executable
>> [ceph02][INFO  ] Running command: sudo /bin/ceph --cluster=ceph osd stat 
>> --format=json
>> [ceph02][INFO  ] Running command: sudo systemctl enable ceph.target
>> [ceph03][DEBUG ] connection detected need for sudo
>> [ceph03][DEBUG ] connected to host: ceph03
>> [ceph03][DEBUG ] detect platform information from remote host
>> [ceph03][DEBUG ] detect machine type
>> [ceph03][DEBUG ] find the location of an executable
>> [ceph_deploy.osd][INFO  ] Distro info: CentOS Linux 7.2.1511 Core
>> [ceph_deploy.osd][DEBUG ] activating host ceph03 disk /dev/sdc
>> [ceph_deploy.osd][DEBUG ] will use init type: systemd
>> [ceph03][DEBUG ] find the location of an executable
>> [ceph03][INFO  ] Running command: sudo /usr/sbin/ceph-disk -v activate 
>> --mark-init systemd --mount /dev/sdc
>> [ceph03][WARNIN] main_activate: path = /dev/sdc
>> [ceph03][WARNIN] No data was received after 300 seconds, disconnecting...
>> [ceph03][INFO  ] checking OSD status...
>> [ceph03][DEBUG ] find the location of an executable
>> [ceph03][INFO  ] Running command: sudo /bin/ceph --cluster=ceph osd stat 
>> --format=json
>> [ceph03][INFO  ] Running command: sudo systemctl enable ceph.target
>>
>> Machines involved are ceph-deploy (deploy server), ceph01 (monitor), ceph02 
>> and
>> ceph03 (OSD servers).
>>
>> ceph log is here:
>>
>> http://pastebin.com/A2kP28c4
>>
>> This is CentOS 5. iptables and selinux are both off. When I first started 
>> doing
>> this the volume would be left mounted in the tmp location on the OSDs. But I
>> have since upgraded my version of ceph and now nothing is left mounted on the
>> OSD but it still times out.
>>
>> Please let me know if there is any other info I can provide which might help.
>> Any help you can offer is greatly appreciated! I've been stuck on this for
>> weeks. Thanks!
>>
>> --
>> Tracy Reed
>
>
>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> --
> Tracy Reed
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Recovery/Backfill Speedup

2016-10-06 Thread Ronny Aasen


how did you set the parameter ?
editing ceph.conf only works when you restart the osd nodes.

but running something like
ceph tell osd.*  injectargs '--osd-max-backfills 6'

would set all osd's max backfill dynamically without restarting the osd. 
and you should fairly quickly afterwards see more backfills in ceph -s



I have also noticed that if i run
ceph -n osd.0 --show-config

on one of my mon nodes, it shows the deafult settings. it does not 
actualy talk to osd.0 and get the current settings. but if i run it from 
any osd node it works. But i am on hammer and not on jewel so this might 
have changed and actualy work for you.



Kind regards
Ronny Aasen


On 05. okt. 2016 21:52, Dan Jakubiec wrote:

Thank Ronny, I am working with Reed on this problem.

Yes something is very strange.  Docs say osd_max_backfills default to
10, but when we examined the run-time configuration using "ceph
--show-config" it was showing osd_max_backfills set to 1 (we are running
latest Jewel release).

We have explicitly set this parameter to 10 now.   Sadly, about 2 hours
in backfills continue to be anemic.   Any other ideas?

$ ceph -s
cluster edeb727e-c6d3-4347-bfbb-b9ce7f60514b
 health HEALTH_WARN
246 pgs backfill_wait
3 pgs backfilling
329 pgs degraded
83 pgs recovery_wait
332 pgs stuck unclean
257 pgs undersized
recovery 154681996/676556815 objects degraded (22.863%)
recovery 278768286/676556815 objects misplaced (41.204%)
noscrub,nodeep-scrub,sortbitwise flag(s) set
 monmap e1: 3 mons at
{core=10.0.1.249:6789/0,db=10.0.1.251:6789/0,dev=10.0.1.250:6789/0}
election epoch 210, quorum 0,1,2 core,dev,db
 osdmap e4274: 16 osds: 16 up, 16 in; 279 remapped pgs
flags noscrub,nodeep-scrub,sortbitwise
  pgmap v1657039: 576 pgs, 2 pools, 6427 GB data, 292 Mobjects
15308 GB used, 101 TB / 116 TB avail
154681996/676556815 objects degraded (22.863%)
278768286/676556815 objects misplaced (41.204%)
 244 active+clean
 242 active+undersized+degraded+remapped+wait_backfill
  53 active+recovery_wait+degraded
  17 active+recovery_wait+degraded+remapped
  13 active+recovery_wait+undersized+degraded+remapped
   3 active+remapped+wait_backfill
   2 active+undersized+degraded+remapped+backfilling
   1 active+degraded+remapped+wait_backfill
   1 active+degraded+remapped+backfilling
recovery io 1568 kB/s, 109 objects/s
  client io 5629 kB/s rd, 411 op/s rd, 0 op/s wr


Here is what our current configuration looks like:

$ ceph -n osd.0 --show-config | grep osd | egrep "recovery|backfill" | sort
osd_allow_recovery_below_min_size = true
osd_backfill_full_ratio = 0.85
osd_backfill_retry_interval = 10
osd_backfill_scan_max = 512
osd_backfill_scan_min = 64
osd_debug_reject_backfill_probability = 0
osd_debug_skip_full_check_in_backfill_reservation = false
osd_kill_backfill_at = 0
osd_max_backfills = 10
osd_min_recovery_priority = 0
osd_recovery_delay_start = 0
osd_recovery_forget_lost_objects = false
osd_recovery_max_active = 15
osd_recovery_max_chunk = 8388608
osd_recovery_max_single_start = 1
osd_recovery_op_priority = 63
osd_recovery_op_warn_multiple = 16
osd_recovery_sleep = 0
osd_recovery_thread_suicide_timeout = 300
osd_recovery_thread_timeout = 30
osd_recovery_threads = 5


-- Dan


Ronny Aasen wrote:

On 04.10.2016 16:31, Reed Dier wrote:

Attempting to expand our small ceph cluster currently.

Have 8 nodes, 3 mons, and went from a single 8TB disk per node to 2x
8TB disks per node, and the rebalancing process is excruciatingly slow.

Originally at 576 PGs before expansion, and wanted to allow rebalance
to finish before expanding the PG count for the single pool, and the
replication size.

I have stopped scrubs for the time being, as well as set client and
recovery io to equal parts so that client io is not burying the
recovery io. Also have increased the number of recovery threads per osd.


[osd]
osd_recovery_threads = 5
filestore_max_sync_interval = 30
osd_client_op_priority = 32
osd_recovery_op_priority = 32

Also, this is 10G networking we are working with and recovery io
typically hovers between 0-35 MB’s but typically very bursty.
Disks are 8TB 7.2k SAS disks behind an LSI 3108 controller,
configured as individual RAID0 VD’s, with pdcache disabled, but BBU
backed write back caching enabled at the controller level.

Have thought about increasing the ‘osd_max_backfills’ as well as
‘osd_recovery_max_active’, and possibly ‘osd_recovery_max_chunk’ to
attempt to speed it up, but will hopefully get some insight from the
community here.

ceph -s about 4 days in:


  health HEALTH_WARN
 255 pgs backfill_wait
 4 pgs backfilling
 385 pgs degraded
 129 pgs recovery_wait

Re: [ceph-users] 6 Node cluster with 24 SSD per node: Hardwareplanning/ agreement

2016-10-06 Thread Denny Fuchs

God morning,

>> * 2 x SN2100 100Gb/s Switch 16 ports
> Which incidentally is a half sized (identical HW really) Arctica 3200C.

really never heart from them :-) (and didn't find any price EUR/$
region) 

>> * 10 x ConnectX 4LX-EN 25Gb card for hypervisor and OSD nodes
 [...] 

> You haven't commented on my rather lengthy mail about your whole design,
> so to reiterate:

maybe accidentally skipped, so much new input  :-) sorry 

> The above will give you a beautiful, fast (but I doubt you'll need the
> bandwidth for your DB transactions), low latency and redundant network
> (these switches do/should support MC-LAG).

Jepp, they do MLAG (with the 25Gbit version of the cx4 NICs) 

> In more technical terms, your network as depicted above can handle under
> normal circumstances around 5GB/s, while your OSD nodes can't write more
> than 1GB/s.
> Massive, wasteful overkill.

before we started with planing Ceph / new hypervisor design, we where
sure that our network would be more powerful, than we need in the near
future. Our applications / DB never used the full 1GBs in any way ... 
we loosing speed on the plain (painful LANCOM) switches and the
applications (mostly Perl written in the beginning of the 2005). 
But anyway, the network should be have enough capacity for the next
years, because it is much more complicated to change network (design)
components, than to kick a node. 

> With a 2nd NVMe in there you'd be at 2GB/s, or simple overkill.

We would buy them ... so that in the end, every 12 disk has a separated
NVMe 

> With decent SSDs and in-line journals (400GB DC S3610s) you'd be at 4.8
> GB/s, a perfect match.

What about the worst case, two nodes are broken, fixed and replaced ? I
red (a lot) that some Ceph users had massive problems, while the rebuild
runs.  

> Of course if your I/O bandwidth needs are actually below 1GB/s at all times
> and all your care about is reducing latency, a single NVMe journal will be
> fine (but also be a very obvious SPoF).

Very happy  to put the finger in the wound, SPof ... is a very hard
thing ... so we try to plan everything redundant  :-) 

The bad side of life: the SSD itself. A consumer SSD lays round about
70/80EUR, a DC SSD jumps up to 120-170EUR. My nightmare is: a lot of
SSDs are jumping over the bridge at the same time  -> arghh  

But, we are working on it :-) 

I've searching an alternative for the Asus board with more PCIe slots
and maybe some components; better CPU with 3.5Ghz-> ; maybe a mix from
the SSDs ... 

At this time, I've found the X10DRi: 

https://www.supermicro.com/products/motherboard/xeon/c600/x10dri.cfm 

and I think we use the E5-2637v4 :-) 

 cu denny___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] offending shards are crashing osd's

2016-10-06 Thread Ronny Aasen


hello

I have a few osd's in my cluster that are regularly crashing.

in the log of them i can see

osd.7
-1> 2016-10-06 08:09:18.869687 7ffaa037f700 -1 osd.7 pg_epoch: 
128840 pg[5.3as0( v 84797'30080 (67219'27080,84797'30080] 
local-les=128834 n=13146 ec=61149 les/c 128834/127358 
128829/128829/128829) [7,109,4,0,62,32]/[7,109,32,0,62,39] r=0 
lpr=128829 pi=127357-128828/12 rops=5 bft=4(2),32(5) crt=0'0 lcod 0'0 
mlcod 0'0 active+remapped+backfilling] handle_recovery_read_complete: 
inconsistent shard sizes 
5/abc6d43a/rbd_data.33640a238e1f29.0003b165/head  the offending 
shard must be manually removed  after verifying there are enough shards 
to recover (0, 8388608, [32(2),0, 39(5),0])



osd.32
  -411> 2016-10-06 13:21:15.166968 7fe45b6cb700 -1 osd.32 pg_epoch: 
129181 pg[5.3as2( v 84797'30080 (67219'27080,84797'30080] 
local-les=129171 n=13146 ec=61149 les/c 129171/127358 
129170/129170/129170) 
[2147483647,2147483647,4,0,62,32]/[2147483647,2147483647,32,0,62,39] r=2 
lpr=129170 pi=121260-129169/43 rops=5 bft=4(2),32(5) crt=0'0 lcod 0'0 
mlcod 0'0 active+undersized+degraded+remapped+backfilling] 
handle_recovery_read_complete: inconsistent shard sizes 
5/abc6d43a/rbd_data.33640a238e1f29.0003b165/head  the offending 
shard must be manually removed  after verifying there are enough shards 
to recover (0, 8388608, [32(2),0, 39(5),0])




osd.109
 -1> 2016-10-06 13:17:36.748340 7fa53d36c700 -1 osd.109 pg_epoch: 
129167 pg[5.3as1( v 84797'30080 (66310'24592,84797'30080] 
local-les=129163 n=13146 ec=61149 les/c 129163/127358 
129162/129162/129162) 
[2147483647,109,4,0,62,32]/[2147483647,109,32,0,62,39] r=1 lpr=129162 
pi=112552-129161/59 rops=5 bft=4(2),32(5) crt=84797'30076 lcod 0'0 mlcod 
0'0 active+undersized+degraded+remapped+backfilling] 
handle_recovery_read_complete: inconsistent shard sizes 
5/abc6d43a/rbd_data.33640a238e1f29.0003b165/head  the offending 
shard must be manually removed  after verifying there are enough shards 
to recover (0, 8388608, [32(2),0, 39(5),0])



ofcourse having 3 osd's dying regularly is not good for my health. so i 
have set noout, to avoid heavy recoveries.


googeling this error messages gives exactly 1 hit:
https://github.com/ceph/ceph/pull/6946

where it saies:  "the shard must be removed so it can be reconstructed"
but with my 3 osd's failing, i am not certain witch of them contain the 
broken shard. (or perhaps all 3 of them?)


a bit reluctant to delete on all 3. I have 4+2 erasure coding.
( erasure size 6 min_size 4 ) so finding out witch one is bad would be 
nice.


hope someone have an idea how to progress.

kind regards
Ronny Aasen

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Multiple storage sites for disaster recovery and/or active-active failover

2016-10-06 Thread Nicolas Huillard

Hello,

I'm new to Ceph, and currently evaluating a deployment strategy.

I'm planning to create a sort of home-hosting (web and compute hosting,
database, etc.), distributed on various locations (cities), extending
the "commodity hardware" concept to "commodity data-center" and
"commodity connectivity". Anything is expected to fail (disks, servers,
switches, routers, Internet fibre links, power, etc.), but the overall
service will still work.
Reaching the services from the outside relies on DNS tuning (add/remove
location when they appears/fail, with a low TTL) and possibly proxies
(providing faster response time by routing traffic through a SPOF...).

Hardware will be recent Xeon D-1500 Mini-ITX motherboards with 2 or 4
Ethernet ports, 6 SATA and 1 NVMe ports, and no PCIe extension cards
(typically Supermicro X10SDV or AsrockRack D1541D4I). A dozen servers
are built in custom enclosures at each location with 12V redundant power
supplies, switches, monitoring, etc.
http://www.supermicro.com/products/motherboard/Xeon/D/X10SDV-F.cfm
http://www.asrockrack.com/general/productdetail.asp?Model=D1541D4I
http://www.pulspower.com/products/show/product/detail/cps20121/

I am wondering how Ceph could provide the location-to-location
fail-over, possibly adding an active-active feature. I'm planning to use
CephFS (shared storage) and RBD (VMs).
I'm not sure yet how to deal with Postgres and other specific services
replication.

Say I have a CephFS at location A, in read-write use at the moment,
serving HTTP requests via the websites/apps/etc. It should replicate
it's data to location B, which could be in standby or read-only mode
(preferably), potentially serving HTTP requests (provided there are no
filesystem writes from those requests). The link between location A and
B (and potentially C, D, etc) is the same Internet fibre link from the
local ISP: not fault-tolerant, subject to lantency, etc.
When the fibre link or power supply fails, the other locations should
notice, change the DNS settings (to disable the requests going to
location A), switch the CephFS to active or read-write mode, and
continue serving requests.
I can handle a few minutes of HTTP downtime, but data should always be
accessible from somewhere, possibly with a few minutes loss but no
crash.

As I read the docs, CephFS and RBD do not handle that situation. RadosGW
have a sort of data replication between clusters and/or pools.
I'm not sure if that problem is solved by the CRUSH rulesets, which
would have to be fine-tuned (say location A is a sort of "room" in the
Crush hierarchy, if I have 2 enclosures with 10 servers in the same
location, those enclosures are "racks", etc.)
Will CRUSH handle latency, failed links, failed power, etc?
How does it solve the CephFS need (active-standby or active-active)?

Nota: I'm also evaluating DRBD, which I know quite well, which have
evolved since my last setup, which does not solve the same low-level
problems, but may also be used in my case, obviously not at the same
scale.

Thanks in advance, for your reading patience and answers!

-- 
Nicolas Huillard
Associé fondateur - Directeur Technique - Dolomède

nhuill...@dolomede.fr
Fixe : +33 9 52 31 06 10
Mobile : +33 6 50 27 69 08
http://www.dolomede.fr/

http://www.350.org/

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Appending to an erasure coded pool

2016-10-06 Thread James Norman

Hi there,

I am developing a web application that supports browsing, uploading, 
downloading, moving files in Ceph Rados pool. Internally to write objects we 
use rados_append, as it's often too memory intensive for us to have the full 
file in memory to do a rados_write_full.

We do not control our customer's Ceph installations, such as whether they use 
replicated pools, EC pools etc. We've found that when dealing with a EC pool, 
our rados_append calls return error code 95 and message "Operation not 
supported".

I've had several discussions with members in the IRC chatroom regarding this, 
and the general consensus I've got is:
1) Use write alignment.
2) Put a replicated pool in front of the EC pool
3) EC pools have a limited feature set

Regarding point 1), are there any actual code example for how you would handle 
this in the context of rados_append? I have struggled to find even one. This 
seems to me something that should be handled by either the API libraries, or 
Ceph itself, not the client trying to write some data.

Regarding point 2) This seems to be a workaround, and generally not something 
we want to recommend to our customers. Is it detrimental to us an EC pool 
without a replicated pool? What are the performance costs of doing so?

Regarding point 3) Can you point me towards resources that describe what 
features / abilities you lose by adopting an EC pool?

Many thanks in advance,

James Norman






signature.asc
Description: Message signed with OpenPGP using GPGMail
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Migrate pre-Jewel RGW data to Jewel realm/zonegroup/zone?

2016-10-06 Thread Richard Chan

Upgraded from Hammer 0.94.9 to Jewel 10.2.3 while all RGW data survived in
a realm-less, setup (no realm, "default" zonegroup, "default" zone).

Is it possible to "move" this into a single realm/zonegroup/zone in
preparation for multisite (i.e before add the 2nd zone).

I don't need more than one realm, but would like to

1. create a single realm"default"
2. move zonegroup "default" as "us", say
3. move zone to zonegroup as "us-east"

This must preserve all existing data. After which I would like to add a 2nd
zone (in a different ceph cluster) "us-west".


# radosgw-admin realm list
{
"default_info": "",
"realms": []
}

# radosgw-admin zonegroup list
read_default_id : -2
{
"default_info": "",
"zonegroups": [
"default"
]
}


# radosgw-admin zone list
{
"default_info": "",
"zones": [
"default"
]
}






-- 
Richard Chan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS: No space left on device

2016-10-06 Thread mykola.dvornik

Is there any way to repair pgs/cephfs gracefully?

-Mykola

From: Yan, Zheng
Sent: Thursday, 6 October 2016 04:48
To: Mykola Dvornik
Cc: John Spray; ceph-users
Subject: Re: [ceph-users] CephFS: No space left on device

On Wed, Oct 5, 2016 at 2:27 PM, Mykola Dvornik  wrote:
> Hi Zheng,
>
> Many thanks for you reply.
>
> This indicates the MDS metadata is corrupted. Did you do any unusual
> operation on the cephfs? (e.g reset journal, create new fs using
> existing metadata pool)
>
> No, nothing has been explicitly done to the MDS. I had a few inconsistent
> PGs that belonged to the (3 replica) metadata pool. The symptoms were
> similar to http://tracker.ceph.com/issues/17177 . The PGs were eventually
> repaired and no data corruption was expected as explained in the ticket.
>

I'm afraid that issue does cause corruption.

> BTW, when I posted this issue on the ML the amount of ground state stry
> objects was around 7.5K. Now it went up to 23K. No inconsistent PGs or any
> other problems happened to the cluster within this time scale.
>
> -Mykola
>
> On 5 October 2016 at 05:49, Yan, Zheng  wrote:
>>
>> On Mon, Oct 3, 2016 at 5:48 AM, Mykola Dvornik 
>> wrote:
>> > Hi Johan,
>> >
>> > Many thanks for your reply. I will try to play with the mds tunables and
>> > report back to your ASAP.
>> >
>> > So far I see that mds log contains a lot of errors of the following
>> > kind:
>> >
>> > 2016-10-02 11:58:03.002769 7f8372d54700  0 mds.0.cache.dir(100056ddecd)
>> > _fetched  badness: got (but i already had) [inode 10005729a77 [2,head]
>> > ~mds0/stray1/10005729a77 auth v67464942 s=196728 nl=0 n(v0 b196728
>> > 1=1+0)
>> > (iversion lock) 0x7f84acae82a0] mode 33204 mtime 2016-08-07
>> > 23:06:29.776298
>> >
>> > 2016-10-02 11:58:03.002789 7f8372d54700 -1 log_channel(cluster) log
>> > [ERR] :
>> > loaded dup inode 10005729a77 [2,head] v68621 at
>> >
>> > /users/mykola/mms/NCSHNO/final/120nm-uniform-h8200/j002654.out/m_xrange192-320_yrange192-320_016232.dump,
>> > but inode 10005729a77.head v67464942 already exists at
>> > ~mds0/stray1/10005729a77
>>
>> This indicates the MDS metadata is corrupted. Did you do any unusual
>> operation on the cephfs? (e.g reset journal, create new fs using
>> existing metadata pool)
>>
>> >
>> > Those folders within mds.0.cache.dir that got badness report a size of
>> > 16EB
>> > on the clients. rm on them fails with 'Directory not empty'.
>> >
>> > As for the "Client failing to respond to cache pressure", I have 2
>> > kernel
>> > clients on 4.4.21, 1 on 4.7.5 and 16 fuse clients always running the
>> > most
>> > recent release version of ceph-fuse. The funny thing is that every
>> > single
>> > client misbehaves from time to time. I am aware of quite discussion
>> > about
>> > this issue on the ML, but cannot really follow how to debug it.
>> >
>> > Regards,
>> >
>> > -Mykola
>> >
>> > On 2 October 2016 at 22:27, John Spray  wrote:
>> >>
>> >> On Sun, Oct 2, 2016 at 11:09 AM, Mykola Dvornik
>> >>  wrote:
>> >> > After upgrading to 10.2.3 we frequently see messages like
>> >>
>> >> From which version did you upgrade?
>> >>
>> >> > 'rm: cannot remove '...': No space left on device
>> >> >
>> >> > The folders we are trying to delete contain approx. 50K files 193 KB
>> >> > each.
>> >>
>> >> My guess would be that you are hitting the new
>> >> mds_bal_fragment_size_max check.  This limits the number of entries
>> >> that the MDS will create in a single directory fragment, to avoid
>> >> overwhelming the OSD with oversized objects.  It is 10 by default.
>> >> This limit also applies to "stray" directories where unlinked files
>> >> are put while they wait to be purged, so you could get into this state
>> >> while doing lots of deletions.  There are ten stray directories that
>> >> get a roughly even share of files, so if you have more than about one
>> >> million files waiting to be purged, you could see this condition.
>> >>
>> >> The "Client failing to respond to cache pressure" messages may play a
>> >> part here -- if you have misbehaving clients then they may cause the
>> >> MDS to delay purging stray files, leading to a backlog.  If your
>> >> clients are by any chance older kernel clients, you should upgrade
>> >> them.  You can also unmount/remount them to clear this state, although
>> >> it will reoccur until the clients are updated (or until the bug is
>> >> fixed, if you're running latest clients already).
>> >>
>> >> The high level counters for strays are part of the default output of
>> >> "ceph daemonperf mds." when run on the MDS server (the "stry" and
>> >> "purg" columns).  You can look at these to watch how fast the MDS is
>> >> clearing out strays.  If your backlog is just because it's not doing
>> >> it fast enough, then you can look at tuning mds_max_purge_files and
>> >> mds_max_purge_ops to adjust the throttles on purging.  Those settings
>> >> can

[ceph-users] SOLVED Re: Can't activate OSD

2016-10-06 Thread Tracy Reed

SOLVED!

Thanks to a very kind person from this list who helped me debug, we found that
when I created the VLAN on the switch I didn't set it allow jumbo packets. This
was preventing the OSDs from activating because some traffic was being blocked.
Once I fixed that everything started working. Sometimes it really helps to have
a second pair of eyes. So this wasn't a Ceph problem at all, really.

Thanks!

On Mon, Oct 03, 2016 at 03:39:45PM PDT, Tracy Reed spake thusly:
> Oops, I said CentOS 5 (old habit, ran it for years!). I meant CentOS 7. And 
> I'm
> running the following Ceph package versions from the ceph repo:
> 
> root@ceph02 ~]# rpm -qa |grep -i ceph
> libcephfs1-10.2.3-0.el7.x86_64
> ceph-common-10.2.3-0.el7.x86_64
> ceph-mon-10.2.3-0.el7.x86_64
> ceph-release-1-1.el7.noarch
> python-cephfs-10.2.3-0.el7.x86_64
> ceph-selinux-10.2.3-0.el7.x86_64
> ceph-osd-10.2.3-0.el7.x86_64
> ceph-mds-10.2.3-0.el7.x86_64
> ceph-radosgw-10.2.3-0.el7.x86_64
> ceph-base-10.2.3-0.el7.x86_64
> ceph-10.2.3-0.el7.x86_64
> 
> On Mon, Oct 03, 2016 at 03:34:50PM PDT, Tracy Reed spake thusly:
> > Hello all,
> > 
> > Over the past few weeks I've been trying to go through the Quick Ceph 
> > Deploy tutorial at:
> > 
> > http://docs.ceph.com/docs/jewel/start/quick-ceph-deploy/
> > 
> > just trying to get a basic 2 OSD ceph cluster up and running. Everything 
> > seems
> > to go well until I get to the:
> > 
> > ceph-deploy osd activate ceph02:/dev/sdc ceph03:/dev/sdc
> > 
> > part. It never actually seems to activate the OSD and eventually times out:
> > 
> > [ceph02][DEBUG ] connection detected need for sudo
> > [ceph02][DEBUG ] connected to host: ceph02 
> > [ceph02][DEBUG ] detect platform information from remote host
> > [ceph02][DEBUG ] detect machine type
> > [ceph02][DEBUG ] find the location of an executable
> > [ceph_deploy.osd][INFO  ] Distro info: CentOS Linux 7.2.1511 Core
> > [ceph_deploy.osd][DEBUG ] activating host ceph02 disk /dev/sdc
> > [ceph_deploy.osd][DEBUG ] will use init type: systemd
> > [ceph02][DEBUG ] find the location of an executable
> > [ceph02][INFO  ] Running command: sudo /usr/sbin/ceph-disk -v activate 
> > --mark-init systemd --mount /dev/sdc
> > [ceph02][WARNIN] main_activate: path = /dev/sdc
> > [ceph02][WARNIN] No data was received after 300 seconds, disconnecting...
> > [ceph02][INFO  ] checking OSD status...
> > [ceph02][DEBUG ] find the location of an executable
> > [ceph02][INFO  ] Running command: sudo /bin/ceph --cluster=ceph osd stat 
> > --format=json
> > [ceph02][INFO  ] Running command: sudo systemctl enable ceph.target
> > [ceph03][DEBUG ] connection detected need for sudo
> > [ceph03][DEBUG ] connected to host: ceph03 
> > [ceph03][DEBUG ] detect platform information from remote host
> > [ceph03][DEBUG ] detect machine type
> > [ceph03][DEBUG ] find the location of an executable
> > [ceph_deploy.osd][INFO  ] Distro info: CentOS Linux 7.2.1511 Core
> > [ceph_deploy.osd][DEBUG ] activating host ceph03 disk /dev/sdc
> > [ceph_deploy.osd][DEBUG ] will use init type: systemd
> > [ceph03][DEBUG ] find the location of an executable
> > [ceph03][INFO  ] Running command: sudo /usr/sbin/ceph-disk -v activate 
> > --mark-init systemd --mount /dev/sdc
> > [ceph03][WARNIN] main_activate: path = /dev/sdc
> > [ceph03][WARNIN] No data was received after 300 seconds, disconnecting...
> > [ceph03][INFO  ] checking OSD status...
> > [ceph03][DEBUG ] find the location of an executable
> > [ceph03][INFO  ] Running command: sudo /bin/ceph --cluster=ceph osd stat 
> > --format=json
> > [ceph03][INFO  ] Running command: sudo systemctl enable ceph.target
> > 
> > Machines involved are ceph-deploy (deploy server), ceph01 (monitor), ceph02 
> > and
> > ceph03 (OSD servers).
> > 
> > ceph log is here:
> > 
> > http://pastebin.com/A2kP28c4
> > 
> > This is CentOS 5. iptables and selinux are both off. When I first started 
> > doing
> > this the volume would be left mounted in the tmp location on the OSDs. But I
> > have since upgraded my version of ceph and now nothing is left mounted on 
> > the
> > OSD but it still times out.
> > 
> > Please let me know if there is any other info I can provide which might 
> > help.
> > Any help you can offer is greatly appreciated! I've been stuck on this for
> > weeks. Thanks!
> > 
> > -- 
> > Tracy Reed
> 
> 
> 
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> -- 
> Tracy Reed



> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


-- 
Tracy Reed


pgpVN3wp3MUC4.pgp
Description: PGP signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph + VMWare

2016-10-06 Thread Daniel Schwager

Hi all,

we are using Ceph (jewel 10.2.2, 10GBit Ceph frontend/backend, 3 nodes, each 8 
OSD's and 2 journal SSD's) 
in out VMware environment especially for test environments and templates - but 
currently 
not for productive machines (because of missing FC-redundancy & performance).

On our Linux based SCST 4GBit fiber channel proxy, 16 ceph-rbd  devices 
(non-caching, in total 10 TB) 
creating a LVM (stripped) volume which is published as a FC-target to our 
VMware cluster. 
Looks fine, works stable. But currently the proxy is not redundant (only one 
head).
Performance is ok (a), but not that good than our IBM Storwize 3700 SAN (16 
HDD's).
Especially for small IO's (4k), the IBM is twice as fast as Ceph. 

Native ceph integration to VMware would be great (-:

Best regards
Daniel

(a) Atto Benchmark screenshots - IBM Storwize 37000 vs. Ceph
https://dtnet.storage.dtnetcloud.com/d/684b330eea/

---
DT Netsolution GmbH   -   Taläckerstr. 30-D-70437 Stuttgart
Geschäftsführer: Daniel Schwager, Stefan Hörz - HRB Stuttgart 19870
Tel: +49-711-849910-32, Fax: -932 - Mailto:daniel.schwa...@dtnet.de

> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of 
> Patrick McGarry
> Sent: Wednesday, October 05, 2016 8:33 PM
> To: Ceph-User; Ceph Devel
> Subject: [ceph-users] Ceph + VMWare
> 
> Hey guys,
> 
> Starting to buckle down a bit in looking at how we can better set up
> Ceph for VMWare integration, but I need a little info/help from you
> folks.
> 
> If you currently are using Ceph+VMWare, or are exploring the option,
> I'd like some simple info from you:
> 
> 1) Company
> 2) Current deployment size
> 3) Expected deployment growth
> 4) Integration method (or desired method) ex: iscsi, native, etc
> 
> Just casting the net so we know who is interested and might want to
> help us shape and/or test things in the future if we can make it
> better. Thanks.
> 


smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] creating a rbd

Re: [ceph-users] creating a rbd

[ceph-users] Hammer OSD memory usage very high

Re: [ceph-users] creating a rbd

Re: [ceph-users] creating a rbd

Re: [ceph-users] creating a rbd

Re: [ceph-users] creating a rbd

[ceph-users] creating a rbd

Re: [ceph-users] jewel/CephFS - misc problems (duplicate strays, mismatch between head items and fnode.fragst)

[ceph-users] jewel/CephFS - misc problems (duplicate strays, mismatch between head items and fnode.fragst)

Re: [ceph-users] unable to start radosgw after upgrade from 10.2.2 to 10.2.3

Re: [ceph-users] Migrate pre-Jewel RGW data to Jewel realm/zonegroup/zone?

Re: [ceph-users] unable to start radosgw after upgrade from 10.2.2 to 10.2.3

Re: [ceph-users] upgrade from v0.94.6 or lower and 'failed to encode map X with expected crc'

Re: [ceph-users] Problem copying a file with ceph-fuse

Re: [ceph-users] Appending to an erasure coded pool

Re: [ceph-users] Migrate pre-Jewel RGW data to Jewel realm/zonegroup/zone?

Re: [ceph-users] Problem copying a file with ceph-fuse

Re: [ceph-users] Problem copying a file with ceph-fuse

[ceph-users] Problem copying a file with ceph-fuse

Re: [ceph-users] Ceph + VMWare

Re: [ceph-users] Can't activate OSD

Re: [ceph-users] Recovery/Backfill Speedup

Re: [ceph-users] 6 Node cluster with 24 SSD per node: Hardwareplanning/ agreement

[ceph-users] offending shards are crashing osd's

[ceph-users] Multiple storage sites for disaster recovery and/or active-active failover

[ceph-users] Appending to an erasure coded pool

[ceph-users] Migrate pre-Jewel RGW data to Jewel realm/zonegroup/zone?

Re: [ceph-users] CephFS: No space left on device

[ceph-users] SOLVED Re: Can't activate OSD

Re: [ceph-users] Ceph + VMWare

31 matches

Site Navigation

Mail list logo

Footer information