Re: [ceph-users] [Nfs-ganesha-devel] 2.7.3 with CEPH_FSAL Crashing

[Daniel Gryniewicz]

ient_request(client.16140784:5570649 mknod
#0x1000907442e/filePathEditorRegistryPrefs.mel3HZLNE 2019-07-15
14:59:44.322408 caller_uid=1161, caller_gid=1131{}) currently
failed to wrlock, waiting
 > > 2019-07-15 15:07:03.014108 7f5fdcdc0700  1 mds.mds01
Updating MDS map to version 68167 from mon.2
 > > 2019-07-15 15:07:04.624096 7f5fda5bb700  0
log_channel(cluster) log [WRN] : 8 slow requests, 1 included
below; oldest blocked for > 44.588632 secs
 > > 2019-07-15 15:07:04.624103 7f5fda5bb700  0
log_channel(cluster) log [WRN] : slow request 34.904077 seconds
old, received at 2019-07-15 15:06:29.719983:
client_request(client.16129440:1072228 getattr pAsLsXsFs
#0x1000907443b 2019-07-15 15:00:03.378512 caller_uid=1161,

caller_gid=1131{1131,4121,2330,2683,4115,2322,2779,2979,1503,3511,2783,2707,2942,2980,2258,2829,1238,1237,2793,1235,1249,2097,1154,2982,2983,3860,4101,1208,3638,3641,3644,3640,3643,3639,3642,3822,3945,4045,3521,3522,3520,3523,})
currently failed to rdlock, waiting
 > > 2019-07-15 15:07:07.013972 7f5fdcdc0700  1 mds.mds01
Updating MDS map to version 68168 from mon.2
 > > 2019-07-15 15:07:09.624166 7f5fda5bb700  0
log_channel(cluster) log [WRN] : 10 slow requests, 2 included
below; oldest blocked for > 49.588693 secs
 > > 2019-07-15 15:07:09.624173 7f5fda5bb700  0
log_channel(cluster) log [WRN] : slow request 32.689838 seconds
old, received at 2019-07-15 15:06:36.934283:
client_request(client.16129440:1072271 getattr pAsLsXsFs
#0x1000907443b 2019-07-15 15:00:10.592734 caller_uid=1161,

caller_gid=1131{1131,4121,2330,2683,4115,2322,2779,2979,1503,3511,2783,2707,2942,2980,2258,2829,1238,1237,2793,1235,1249,2097,1154,2982,2983,3860,4101,1208,3638,3641,3644,3640,3643,3639,3642,3822,3945,4045,3521,3522,3520,3523,})
currently failed to rdlock, waiting
 > > 2019-07-15 15:07:09.624177 7f5fda5bb700  0
log_channel(cluster) log [WRN] : slow request 34.962719 seconds
old, received at 2019-07-15 15:06:34.661402:
client_request(client.16129440:1072256 getattr pAsLsXsFs
#0x1000907443b 2019-07-15 15:00:08.319912 caller_uid=1161,

caller_gid=1131{1131,4121,2330,2683,4115,2322,2779,2979,1503,3511,2783,2707,2942,2980,2258,2829,1238,1237,2793,1235,1249,2097,1154,2982,2983,3860,4101,1208,3638,3641,3644,3640,3643,3639,3642,3822,3945,4045,3521,3522,3520,3523,})
currently failed to rdlock, waiting
 > > 2019-07-15 15:07:11.519928 7f5fdcdc0700  1 mds.mds01
Updating MDS map to version 68169 from mon.2
 > > 2019-07-15 15:07:19.624272 7f5fda5bb700  0
log_channel(cluster) log [WRN] : 11 slow requests, 1 included
below; oldest blocked for > 59.588812 secs
 > > 2019-07-15 15:07:19.624278 7f5fda5bb700  0
log_channel(cluster) log [WRN] : slow request 32.164260 seconds
old, received at 2019-07-15 15:06:47.459980:
client_request(client.16129440:1072326 getattr pAsLsXsFs
#0x1000907443b 2019-07-15 15:00:21.118372 caller_uid=1161,

caller_gid=1131{1131,4121,2330,2683,4115,2322,2779,2979,1503,3511,2783,2707,2942,2980,2258,2829,1238,1237,2793,1235,1249,2097,1154,2982,2983,3860,4101,1208,3638,3641,3644,3640,3643,3639,3642,3822,3945,4045,3521,3522,3520,3523,})
currently failed to rdlock, waiting
 > >
 > >
 > > On Tue, Jul 16, 2019 at 1:18 PM Daniel Gryniewicz
mailto:d...@redhat.com>> wrote:
 > > > This is not one I've seen before, and a quick look at the
code looks
 > > > strange.  The only assert in that bit is asserting the
parent is a
 > > > directory, but the parent directory is not something that
was passed in
 > > > by Ganesha, but rather something that was looked up
internally in
 > > > libcephfs.  This is beyond my expertise, at this point. 
Maybe some ceph

 > > > logs would help?
 > > >
 > > > Daniel
 > > >
 > > > On 7/15/19 10:54 AM, David C wrote:
 > > > > This list has been deprecated. Please subscribe to the
new devel list at lists.nfs-ganesha.org
<http://lists.nfs-ganesha.org>.
 > > > >
 > > > >
 > > > > Hi All
 > > > >
 > > > > I'm running 2.7.3 using the CEPH FSAL to export CephFS
(Luminous), it
 > > > > ran well for a few days and crashed. I have a coredump,
could someone
 > > > > assist me in debugging this please?
 > >

Re: [ceph-users] NFS

[Daniel Gryniewicz]

"Path" is either "/" to indicate the top of the tree, or a bucket name
to indicate a limited export for a single bucket.  It's not related to
the user at all.

On Thu, Oct 3, 2019 at 10:34 AM Marc Roos  wrote:
>
>
> How should a multi tenant RGW config look like, I am not able get this
> working:
>
> EXPORT {
>Export_ID=301;
>Path = "test:test3";
>#Path = "/";
>Pseudo = "/rgwtester";
>
>Protocols = 4;
>FSAL {
>Name = RGW;
>User_Id = "test$tester1";
>Access_Key_Id = "TESTER";
>Secret_Access_Key = "xxx";
>}
>Disable_ACL = TRUE;
>CLIENT { Clients = 192.168.10.0/24; access_type = "RO"; }
> }
>
>
> 03/10/2019 16:15:37 : epoch 5d8d274c : c01 : ganesha.nfsd-4722[sigmgr]
> create_export :FSAL :CRIT :RGW module: librgw init failed (-5)
> 03/10/2019 16:15:37 : epoch 5d8d274c : c01 : ganesha.nfsd-4722[sigmgr]
> mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on
> underlying FSAL RGW
> 03/10/2019 16:15:37 : epoch 5d8d274c : c01 : ganesha.nfsd-4722[sigmgr]
> fsal_put :FSAL :INFO :FSAL RGW now unused
> 03/10/2019 16:15:37 : epoch 5d8d274c : c01 : ganesha.nfsd-4722[sigmgr]
> fsal_cfg_commit :CONFIG :CRIT :Could not create export for (/rgwtester)
> to (test:test3)
> 03/10/2019 16:15:37 : epoch 5d8d274c : c01 : ganesha.nfsd-4722[sigmgr]
> fsal_cfg_commit :FSAL :F_DBG :FSAL RGW refcount 0
> 03/10/2019 16:15:37 : epoch 5d8d274c : c01 : ganesha.nfsd-4722[sigmgr]
> config_errs_to_log :CONFIG :CRIT :Config File
> (/etc/ganesha/ganesha.conf:216): 1 validation errors in block FSAL
> 03/10/2019 16:15:37 : epoch 5d8d274c : c01 : ganesha.nfsd-4722[sigmgr]
> config_errs_to_log :CONFIG :CRIT :Config File
> (/etc/ganesha/ganesha.conf:216): Errors processing block (FSAL)
> 03/10/2019 16:15:37 : epoch 5d8d274c : c01 : ganesha.nfsd-4722[sigmgr]
> config_errs_to_log :CONFIG :CRIT :Config File
> (/etc/ganesha/ganesha.conf:209): 1 validation errors in block EXPORT
> 03/10/2019 16:15:37 : epoch 5d8d274c : c01 : ganesha.nfsd-4722[sigmgr]
> config_errs_to_log :CONFIG :CRIT :Config File
> (/etc/ganesha/ganesha.conf:209): Errors processing block (EXPORT)
>
> -Original Message-
> Subject: Re: [ceph-users] NFS
>
> RGW NFS can support any NFS style of authentication, but users will have
> the RGW access of their nfs-ganesha export.  You can create exports with
> disjoint privileges, and since recent L, N, RGW tenants.
>
> Matt
>
> On Tue, Oct 1, 2019 at 8:31 AM Marc Roos 
> wrote:
> >
> >  I think you can run into problems
> > with a multi user environment of RGW and nfs-ganesha.
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> --
>
> Matt Benjamin
> Red Hat, Inc.
> 315 West Huron Street, Suite 140A
> Ann Arbor, Michigan 48103
>
> http://www.redhat.com/en/technologies/storage
>
> tel.  734-821-5101
> fax.  734-769-8938
> cel.  734-216-5309
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] NFS

[Daniel Gryniewicz]

So, Ganesha is an NFS gateway, living in userspace.  It provides
access via NFS (for any NFS client) to a number of clustered storage
systems, or to local filesystems on it's host.  It can run on any
system that has access to the cluster (ceph in this case).  One
Ganesha instance can serve quite a few clients (the limit typically
being either memory on the Ganesha node or network bandwidth).
Ganesha's configuration lives in /etc/ganesha/ganesha.conf.  There
should be man pages related to Ganesha and it's configuration
installed when Ganesha is installed.

Ganesha has a number of FSALs (File System Abstraction Layers) that
work with a number of different clustered storage systems.  For Ceph,
Ganesha has 2 FSALs: FSAL_CEPH works on top of CephFS, and FSAL_RGW
works on top of RadosGW.  FSAL_CEPH provides full NFS semantics,
sinces CephFS is a full POSIX filesystem; FSAL_RGW provides slightly
limited semantics, since RGW itself it not POSIX and doesn't provide
everything.  For example, you cannot write into an arbitrary location
within a file, you can only overwrite the entire file.

Anything you can store in the underlying storage (CephFS or RadosGW)
can be stored/accessed by Ganesha.  So, 20+GB files should work fine
on either one.

Daniel

On Tue, Oct 1, 2019 at 10:45 PM Brent Kennedy  wrote:
>
> We might have to backup a step here so I can understand.  Are you saying 
> stand up a new VM with just those packages installed, then configure the 
> export file  ( the file location isn’t mentioned in the ceph docs ) and 
> supposedly a client can connect to them?  ( only linux clients or any NFS 
> client? )
>
> I don’t use cephFS, so being that it will be an object storage backend, will 
> that be ok with multiple hosts accessing files through the NFS one gateway or 
> should I configure multiple gateways ( one for each share )?
>
> I was hoping to save large files( 20+ GB ), should I stand up cephFS instead 
> for this?
>
> I am used to using a NAS storage appliance server(or freeNAS ), so using ceph 
> as a NAS backend is new to me ( thus I might be over thinking this )  :)
>
> -Brent
>
> -Original Message-
> From: Daniel Gryniewicz 
> Sent: Tuesday, October 1, 2019 8:20 AM
> To: Marc Roos ; bkennedy ; 
> ceph-users 
> Subject: Re: [ceph-users] NFS
>
> Ganesha can export CephFS or RGW.  It cannot export anything else (like iscsi 
> or RBD).  Config for RGW looks like this:
>
> EXPORT
> {
>  Export_ID=1;
>  Path = "/";
>  Pseudo = "/rgw";
>  Access_Type = RW;
>  Protocols = 4;
>  Transports = TCP;
>  FSAL {
>  Name = RGW;
>  User_Id = "testuser";
>  Access_Key_Id ="";
>  Secret_Access_Key = "";
>  }
> }
>
> RGW {
>  ceph_conf = "//ceph.conf";
>  # for vstart cluster, name = "client.admin"
>  name = "client.rgw.foohost";
>  cluster = "ceph";
> #   init_args = "-d --debug-rgw=16";
> }
>
>
> Daniel
>
> On 9/30/19 3:01 PM, Marc Roos wrote:
> >
> > Just install these
> >
> > http://download.ceph.com/nfs-ganesha/
> > nfs-ganesha-rgw-2.7.1-0.1.el7.x86_64
> > nfs-ganesha-vfs-2.7.1-0.1.el7.x86_64
> > libnfsidmap-0.25-19.el7.x86_64
> > nfs-ganesha-mem-2.7.1-0.1.el7.x86_64
> > nfs-ganesha-xfs-2.7.1-0.1.el7.x86_64
> > nfs-ganesha-2.7.1-0.1.el7.x86_64
> > nfs-ganesha-ceph-2.7.1-0.1.el7.x86_64
> >
> >
> > And export your cephfs like this:
> > EXPORT {
> >  Export_Id = 10;
> >  Path = /nfs/cblr-repos;
> >  Pseudo = /cblr-repos;
> >  FSAL { Name = CEPH; User_Id = "cephfs.nfs.cblr";
> > Secret_Access_Key = "xxx"; }
> >  Disable_ACL = FALSE;
> >  CLIENT { Clients = 192.168.10.2; access_type = "RW"; }
> >  CLIENT { Clients = 192.168.10.253; } }
> >
> >
> > -Original Message-
> > From: Brent Kennedy [mailto:bkenn...@cfl.rr.com]
> > Sent: maandag 30 september 2019 20:56
> > To: 'ceph-users'
> > Subject: [ceph-users] NFS
> >
> > Wondering if there are any documents for standing up NFS with an
> > existing ceph cluster.  We don’t use ceph-ansible or any other tools
> > besides ceph-deploy.  The iscsi directions were pretty good once I got
> > past the dependencies.
> >
> >
> >
> > I saw the one based on Rook, but it doesn’t seem to apply to our setup
> > of ceph vms with physical hosts doing OSDs.  The official ceph
> > documents talk about using ganesha but 

Re: [ceph-users] NFS

[Daniel Gryniewicz]

Ganesha can export CephFS or RGW.  It cannot export anything else (like 
iscsi or RBD).  Config for RGW looks like this:


EXPORT
{
Export_ID=1;
Path = "/";
Pseudo = "/rgw";
Access_Type = RW;
Protocols = 4;
Transports = TCP;
FSAL {
Name = RGW;
User_Id = "testuser";
Access_Key_Id ="";
Secret_Access_Key = "";
}
}

RGW {
ceph_conf = "//ceph.conf";
# for vstart cluster, name = "client.admin"
name = "client.rgw.foohost";
cluster = "ceph";
#   init_args = "-d --debug-rgw=16";
}


Daniel

On 9/30/19 3:01 PM, Marc Roos wrote:
  
Just install these


http://download.ceph.com/nfs-ganesha/
nfs-ganesha-rgw-2.7.1-0.1.el7.x86_64
nfs-ganesha-vfs-2.7.1-0.1.el7.x86_64
libnfsidmap-0.25-19.el7.x86_64
nfs-ganesha-mem-2.7.1-0.1.el7.x86_64
nfs-ganesha-xfs-2.7.1-0.1.el7.x86_64
nfs-ganesha-2.7.1-0.1.el7.x86_64
nfs-ganesha-ceph-2.7.1-0.1.el7.x86_64


And export your cephfs like this:
EXPORT {
 Export_Id = 10;
 Path = /nfs/cblr-repos;
 Pseudo = /cblr-repos;
 FSAL { Name = CEPH; User_Id = "cephfs.nfs.cblr";
Secret_Access_Key = "xxx"; }
 Disable_ACL = FALSE;
 CLIENT { Clients = 192.168.10.2; access_type = "RW"; }
 CLIENT { Clients = 192.168.10.253; }
}


-Original Message-
From: Brent Kennedy [mailto:bkenn...@cfl.rr.com]
Sent: maandag 30 september 2019 20:56
To: 'ceph-users'
Subject: [ceph-users] NFS

Wondering if there are any documents for standing up NFS with an
existing ceph cluster.  We don’t use ceph-ansible or any other tools
besides ceph-deploy.  The iscsi directions were pretty good once I got
past the dependencies.

  


I saw the one based on Rook, but it doesn’t seem to apply to our setup
of ceph vms with physical hosts doing OSDs.  The official ceph documents
talk about using ganesha but doesn’t seem to dive into the details of
what the process is for getting it online.  We don’t use cephfs, so
that’s not setup either.  The basic docs seem to note this is required.
  Seems my google-fu is failing me when I try to find a more definitive
guide.

  


The servers are all centos 7 with the latest updates.

  


Any guidance would be greatly appreciated!

  


Regards,

-Brent

  


Existing Clusters:

Test: Nautilus 14.2.2 with 3 osd servers, 1 mon/man, 1 gateway, 2 iscsi
gateways ( all virtual on nvme )

US Production(HDD): Nautilus 14.2.2 with 13 osd servers, 3 mons, 4
gateways, 2 iscsi gateways

UK Production(HDD): Nautilus 14.2.2 with 25 osd servers, 3 mons/man, 3
gateways behind

US Production(SSD): Nautilus 14.2.2 with 6 osd servers, 3 mons/man, 3
gateways, 2 iscsi gateways

  



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph nfs ganesha exports

[Daniel Gryniewicz]

User credentials in NFS are in the form of UID/GID pairs, just like in 
POSIX.  These creds are specified by the NFS client, possibly modified 
by the server using NFS user mapping, and then passed down to CephFS by 
Ganesha.  Whether or not the client user has permissions to do a 
particular op (for example, write) depends on whether CephFS allows that 
UID/GID pair to do the operation.  This is primarily controlled by mode, 
but also by ACLs on the file/directory.


This is orthogonal to the Ceph user ID that Ganesha uses to connect, 
except that the permissions of that user are the permissions used for 
the "root" UID, so you can't have more permissions than that user grants.


Daniel

On 7/28/19 2:20 PM, Lee Norvall wrote:
Update to this I found that you cannot create a 2nd files system as yet 
and it is still experimental.  So I went down this route:


Added a pool to the existing cephfs and then setfattr -n 
ceph.dir.layout.pool -v SSD-NFS /mnt/cephfs/ssdnfs/ from a ceph-fuse client.


I then nfs mounted from another box. I can see the files and dir etc 
from the nfs client but my issue now is that I do not have permission to 
write, create dir etc.  The same goes for the default setup after 
running the ansible playbook even when setting export to 
no_root_squash.  I am missing a chain of permission?  ganesha-nfs is 
using admin userid, is this the same as the client.admin or is this a 
user I need to create?  Any info appreciated.


Ceph is on CentOS 7 and SELinux is currently off as well.

Copy of the ganesha conf below.  Is secType correct or is it missing 
something?


RADOS_URLS {
    ceph_conf = '/etc/ceph/ceph.conf';
    userid = "admin";
}
%url rados://cephfs_data/ganesha-export-index

NFSv4 {
     RecoveryBackend = 'rados_kv';
}
RADOS_KV {
     ceph_conf = '/etc/ceph/ceph.conf';
     userid = "admin";
     pool = "cephfs_data";
}

EXPORT
{
     Export_id=20133;
     Path = "/";
     Pseudo = /cephfile;
     Access_Type = RW;
     Protocols = 3,4;
     Transports = TCP;
     SecType = sys,krb5,krb5i,krb5p;
     Squash = Root_Squash;
     Attr_Expiration_Time = 0;

     FSAL {
     Name = CEPH;
     User_Id = "admin";
     }


}





On 28/07/2019 12:11, Lee Norvall wrote:

Hi

I am using ceph-ansible to deploy and just looking for best way/tips on
how to export multiple pools/fs.

Ceph: nautilus (14.2.2)
NFS-Ganesha v 2.8
ceph-ansible stable 4.0

I have 3 x osd/NFS gateways running and NFS on the dashboard can see
them in the cluster.  I have managed to export for cephfs / and mounted
it on another box.

1) can I add a new pool/fs to the export under that same NFS gateway
cluster, or

2) do I have the to do something like add a new pool to the fs and then
setfattr to make the layout /newfs_dir point to /new_pool?  does this
cause issues and false object count?

3) any other better ways...

Rgds

Lee

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

--
blocz IO Limted

*Lee Norvall | CEO / Founder*
*Mob.* +44 (0)7768 201884
*Tel.* +44 (0)20 3026 8930
*Web.* www.blocz.io

*Enterprise Cloud | Private Cloud | Hybrid/Multi Cloud | Cloud Backup *

blocz IO Limted blocz IO Limted blocz IO Limted blocz IO Limted blocz IO 
Limted ESET MSP Partner


This e-mail (and any attachment) has been sent from a PC belonging to My 
Mind (Holdings) Limited. If you receive it in error, please tell us by 
return and then delete it from your system; you may not rely on its 
contents nor copy/disclose it to anyone. Opinions, conclusions and 
statements of intent in this e-mail are those of the sender and will not 
bind My Mind (Holdings) Limited unless confirmed by an authorised 
representative independently of this message. We do not accept 
responsibility for viruses; you must scan for these. Please note that 
e-mails sent to and from blocz IO Limited are routinely monitored for 
record keeping, quality control and training purposes, to ensure 
regulatory compliance and to prevent viruses and unauthorised use of our 
computer systems. My Mind (Holdings) Limited is registered in England & 
Wales under company number 10186410. Registered office: 1st Floor 
Offices, 2a Highfield Road, Ringwood, Hampshire, United Kingdom, BH24 
1RQ. VAT Registration GB 244 9628 77


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW Beast frontend and ipv6 options

[Daniel Gryniewicz]

After discussing with Casey, I'd like to propose some clarifications to 
this.


First, we do not treat EAFNOSUPPORT as a non-fatal error.  Any other 
error binding is fatal, but that one we warn and continue.


Second, we treat "port=" as expanding to "endpoint=0.0.0.0:, 
endpoint=[::]".


Then, we just process the set of endpoints properly.  This should, I 
believe, result in simple, straight-forward code, and easily 
understandable semantics, and should make it simple for orchetrators.


This would make 1 and 2 below fallout naturally.  3 is modified so that 
we only use configured endpoints, but port= is now implicit endpoint 
configuration.


Daniel

On 5/2/19 10:08 AM, Daniel Gryniewicz wrote:
Based on past experience with this issue in other projects, I would 
propose this:


1. By default (rgw frontends=beast), we should bind to both IPv4 and 
IPv6, if available.


2. Just specifying port (rgw frontends=beast port=8000) should apply to 
both IPv4 and IPv6, if available.


3. If the user provides endpoint config, we should use only that 
endpoint config.  For example, if they provide only v4 addresses, we 
should only bind to v4.


This should all be independent of the bindv6only setting; that is, we 
should specifically bind our v4 and v6 addresses, and not depend on the 
system to automatically bind v4 when binding v6.


In the case of 1 or 2, if the system has disabled either v4 or v6, this 
should not be an error, as long as one of the two binds works.  In the 
case of 3, we should error out if any configured endpoint cannot be bound.


This should allow an orchestrator to confidently install a system, 
knowing what will happen, without needing to know or manipulate the 
bindv6only flag.


As for what happens if you specify an endpoint and a port, I don't have 
a strong opinion.  I see 2 reasonable possibilites:


a. Make it an error

b. Treat a port in this case as an endpoint of 0.0.0.0:port (v4-only)

Daniel

On 4/26/19 4:49 AM, Abhishek Lekshmanan wrote:


Currently RGW's beast frontend supports ipv6 via the endpoint
configurable. The port option will bind to ipv4 _only_.

http://docs.ceph.com/docs/master/radosgw/frontends/#options

Since many Linux systems may default the sysconfig net.ipv6.bindv6only
flag to true, it usually means that specifying a v6 endpoint will bind
to both v4 and v6. But this also means that deployment systems must be
aware of this while configuring depending on whether both v4 and v6
endpoints need to work or not. Specifying both a v4 and v6 endpoint or a
port (v4) and endpoint with the same v6 port will currently lead to a
failure as the system would've already bound the v6 port to both v4 and
v6. This leaves us with a few options.

1. Keep the implicit behaviour as it is, document this, as systems are
already aware of sysconfig flags and will expect that at a v6 endpoint
will bind to both v4 and v6.

2. Be explicit with endpoints & configuration, Beast itself overrides
the socket option to bind both v4 and v6, which means that v6 endpoint
will bind to v6 *only* and binding to a v4 will need an explicit
specification. (there is a pr in progress for this:
https://github.com/ceph/ceph/pull/27270)

Any more suggestions on how systems handle this are also welcome.

--
Abhishek
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com





___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW Beast frontend and ipv6 options

[Daniel Gryniewicz]

Based on past experience with this issue in other projects, I would 
propose this:


1. By default (rgw frontends=beast), we should bind to both IPv4 and 
IPv6, if available.


2. Just specifying port (rgw frontends=beast port=8000) should apply to 
both IPv4 and IPv6, if available.


3. If the user provides endpoint config, we should use only that 
endpoint config.  For example, if they provide only v4 addresses, we 
should only bind to v4.


This should all be independent of the bindv6only setting; that is, we 
should specifically bind our v4 and v6 addresses, and not depend on the 
system to automatically bind v4 when binding v6.


In the case of 1 or 2, if the system has disabled either v4 or v6, this 
should not be an error, as long as one of the two binds works.  In the 
case of 3, we should error out if any configured endpoint cannot be bound.


This should allow an orchestrator to confidently install a system, 
knowing what will happen, without needing to know or manipulate the 
bindv6only flag.


As for what happens if you specify an endpoint and a port, I don't have 
a strong opinion.  I see 2 reasonable possibilites:


a. Make it an error

b. Treat a port in this case as an endpoint of 0.0.0.0:port (v4-only)

Daniel

On 4/26/19 4:49 AM, Abhishek Lekshmanan wrote:


Currently RGW's beast frontend supports ipv6 via the endpoint
configurable. The port option will bind to ipv4 _only_.

http://docs.ceph.com/docs/master/radosgw/frontends/#options

Since many Linux systems may default the sysconfig net.ipv6.bindv6only
flag to true, it usually means that specifying a v6 endpoint will bind
to both v4 and v6. But this also means that deployment systems must be
aware of this while configuring depending on whether both v4 and v6
endpoints need to work or not. Specifying both a v4 and v6 endpoint or a
port (v4) and endpoint with the same v6 port will currently lead to a
failure as the system would've already bound the v6 port to both v4 and
v6. This leaves us with a few options.

1. Keep the implicit behaviour as it is, document this, as systems are
already aware of sysconfig flags and will expect that at a v6 endpoint
will bind to both v4 and v6.

2. Be explicit with endpoints & configuration, Beast itself overrides
the socket option to bind both v4 and v6, which means that v6 endpoint
will bind to v6 *only* and binding to a v4 will need an explicit
specification. (there is a pr in progress for this:
https://github.com/ceph/ceph/pull/27270)

Any more suggestions on how systems handle this are also welcome.

--
Abhishek
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph expansion/deploy via ansible

[Daniel Gryniewicz]

On 4/17/19 4:24 AM, John Molefe wrote:

Hi everyone,

I currently have a ceph cluster running on SUSE and I have an expansion 
project that I will be starting with around June.
Has anybody here deployed (from scratch) or expanded their ceph cluster 
via ansible?? I would appreciate it if you'd share your experiences, 
challenges, topology, etc.




I've done both, many times, although only with test clusters in the 3-8 
machine range.  My experience with ceph-ansible is that initial config 
is finicky, and needs to be gotten exactly right; but that once you have 
a working config, it just works for installs and expansions.  I've added 
machines one at a time and multiples at a time, and it works well.


Daniel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Deploy Cehp in multisite setup

[Daniel Gryniewicz]

On 3/5/19 2:15 PM, Paul Emmerich wrote:

Choose two:

* POSIX filesystem with a reliable storage underneath
* Multiple sites with poor or high-latency connection between them
* Performance



If you can get away with S3/Object access, rather than POSIX FS, you 
could use the RadosGW from Ceph.  It has multisite syncing that is done 
out-of-band from the client.  (That is, if you chose the second and 
third options above.)


Daniel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CEPH_FSAL Nfs-ganesha

[Daniel Gryniewicz]

Hi.  Welcome to the community.

On 01/14/2019 07:56 AM, David C wrote:

Hi All

I've been playing around with the nfs-ganesha 2.7 exporting a cephfs 
filesystem, it seems to be working pretty well so far. A few questions:


1) The docs say " For each NFS-Ganesha export, FSAL_CEPH uses a 
libcephfs client,..." [1]. For arguments sake, if I have ten top level 
dirs in my Cephfs namespace, is there any value in creating a separate 
export for each directory? Will that potentially give me better 
performance than a single export of the entire namespace?


I don't believe there are any advantages from the Ceph side.  From the 
Ganesha side, you configure permissions, client ACLs, squashing, and so 
on on a per-export basis, so you'll need different exports if you need 
different settings for each top level directory.  If they can all use 
the same settings, one export is probably better.




2) Tuning: are there any recommended parameters to tune? So far I've 
found I had to increase client_oc_size which seemed quite conservative.


Ganesha is just a standard libcephfs client, so any tuning you'd make on 
any other cephfs client also applies to Ganesha.  I'm not aware of 
anything in particular, but I've never deployed it for anything other 
than testing.


Daniel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How to troubleshoot rsync to cephfs via nfs-ganesha stalling

[Daniel Gryniewicz]

Okay, this all looks fine, and it's extremely unlikely that a text file 
will have holes in it (I thought holes, because rsync handles holes, but 
wget would just copy zeros instead).


Is this reproducible?  If so, can you turn up Ganesha logging and post a 
log file somewhere?


Daniel

On 12/12/2018 04:56 AM, Marc Roos wrote:
  
Hi Daniel, thanks for looking at this.


These are the mount options
  type nfs4
(rw,nodev,relatime,vers=4,intr,local_lock=none,retrans=2,proto=tcp,rsize
=8192,wsize=8192,hard,namlen=255,sec=sys)

I have overwritten the original files, so I cannot examine if they had
holes. To be honest I don't even know how to query the file, to identify
holes.

These are the contents of the files, just plain text.
[@os0 CentOS7-x86_64]# cat CentOS_BuildTag
20181125-1500
[@os0 CentOS7-x86_64]# cat .discinfo
1543162572.807980
7.6
x86_64



-Original Message-
From: Daniel Gryniewicz [mailto:d...@redhat.com]
Sent: 10 December 2018 15:54
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] How to troubleshoot rsync to cephfs via
nfs-ganesha stalling

This isn't something I've seen before.  rsync generally works fine, even
over cephfs.  More inline.

On 12/09/2018 09:42 AM, Marc Roos wrote:



This rsync command fails and makes the local nfs unavailable (Have to
stop nfs-ganesha, kill all rsync processes on the client and then
start
nfs-ganesha)

rsync -rlptDvSHP --delete  --exclude config.repo --exclude "local*"
--exclude "isos"
anonym...@mirror.ams1.nl.leaseweb.net::centos/7/os/x86_64/
/localpath/CentOS7-x86_64/

When I do individual rsyncs on the subfolders

-rw-r--r-- 1 nobody 500   14 Nov 25 17:01 CentOS_BuildTag
-rw-r--r-- 1 nobody 500   29 Nov 25 17:16 .discinfo
drwxr-xr-x 3 nobody 500 8.3M Nov 25 17:20 EFI
-rw-rw-r-- 1 nobody 500  227 Aug 30  2017 EULA
-rw-rw-r-- 1 nobody 500  18K Dec  9  2015 GPL drwxr-xr-x 3 nobody 500
572M Nov 25 17:21 images drwxr-xr-x 2 nobody 500  57M Dec  9 14:11
isolinux drwxr-xr-x 2 nobody 500 433M Nov 25 17:20 LiveOS drwxrwxr-x 2



nobody 500 9.5G Nov 25 16:58 Packages drwxrwxr-x 2 nobody 500  29M Dec
  

9 13:53 repodata
-rw-rw-r-- 1 nobody 500 1.7K Dec  9  2015 RPM-GPG-KEY-CentOS-7
-rw-rw-r-- 1 nobody 500 1.7K Dec  9  2015 RPM-GPG-KEY-CentOS-Testing-7
-rw-r--r-- 1 nobody 500  354 Nov 25 17:21 .treeinfo

These rsyncs are all going fine.

rsync -rlptDvSHP --delete  --exclude config.repo --exclude "local*"
--exclude "isos"
anonym...@mirror.ams1.nl.leaseweb.net::centos/7/os/x86_64/Packages/
/localpath/CentOS7-x86_64/Packages/
rsync -rlptDvSHP --delete  --exclude config.repo --exclude "local*"
--exclude "isos"
anonym...@mirror.ams1.nl.leaseweb.net::centos/7/os/x86_64/repodata/
/localpath/CentOS7-x86_64/repodata/
rsync -rlptDvSHP --delete  --exclude config.repo --exclude "local*"
--exclude "isos"
anonym...@mirror.ams1.nl.leaseweb.net::centos/7/os/x86_64/LiveOS/
/localpath/CentOS7-x86_64/LiveOS/

Except when I try to rsync the file CentOS_BuildTag then everything
stalls. Leaving such files
-rw--- 1 500 500 0 Dec  9 14:26 .CentOS_BuildTag.2igwc5
-rw--- 1 500 500 0 Dec  9 14:28 .CentOS_BuildTag.tkiwc5


So something is failing on the write, it seems.  These are the temporary
files made by rsync, and they're empty, so the initial write seems to
have failed.


I can resolf this by doing a wget and moving the file to the location
wget


'http://mirror.ams1.nl.leaseweb.net/centos/7/os/x86_64/CentOS_BuildTag'

mv CentOS_BuildTag /localpath/CentOS7-x86_64/

I had also problems with .discinfo and when I ls this directory on
cephfs mount it takes a long time to produce output.

When I do the full rsync to the cephfs mount it completes without
errors, when I then later do the sync on the nfs mount it completes
also (nothing being copied)


This confirms that it's not metadata related, as this second successful
rsync is purely metadata.


Anybody know what I should do to resolv this? Is this a typical
ganesha issue or is this cephfs corruption, that make ganesha stall?


Writes in Ganesha are pretty much passthrough, modulo some metadata
tracking.  This means that a write hang is likely to be somewhere
between Ganesha and CephFS.  However, this is a single, small file, so I
don't see how it could hang, especially when wget can copy the file
correctly.  Maybe there's something about the structure of the file?
Does it have holes in it, for example?

Also, can you send the mount options for the NFS mount?

Daniel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How to troubleshoot rsync to cephfs via nfs-ganesha stalling

[Daniel Gryniewicz]

This isn't something I've seen before.  rsync generally works fine, even 
over cephfs.  More inline.


On 12/09/2018 09:42 AM, Marc Roos wrote:



This rsync command fails and makes the local nfs unavailable (Have to
stop nfs-ganesha, kill all rsync processes on the client and then start
nfs-ganesha)

rsync -rlptDvSHP --delete  --exclude config.repo --exclude "local*"
--exclude "isos"
anonym...@mirror.ams1.nl.leaseweb.net::centos/7/os/x86_64/
/localpath/CentOS7-x86_64/

When I do individual rsyncs on the subfolders

-rw-r--r-- 1 nobody 500   14 Nov 25 17:01 CentOS_BuildTag
-rw-r--r-- 1 nobody 500   29 Nov 25 17:16 .discinfo
drwxr-xr-x 3 nobody 500 8.3M Nov 25 17:20 EFI
-rw-rw-r-- 1 nobody 500  227 Aug 30  2017 EULA
-rw-rw-r-- 1 nobody 500  18K Dec  9  2015 GPL
drwxr-xr-x 3 nobody 500 572M Nov 25 17:21 images
drwxr-xr-x 2 nobody 500  57M Dec  9 14:11 isolinux
drwxr-xr-x 2 nobody 500 433M Nov 25 17:20 LiveOS
drwxrwxr-x 2 nobody 500 9.5G Nov 25 16:58 Packages
drwxrwxr-x 2 nobody 500  29M Dec  9 13:53 repodata
-rw-rw-r-- 1 nobody 500 1.7K Dec  9  2015 RPM-GPG-KEY-CentOS-7
-rw-rw-r-- 1 nobody 500 1.7K Dec  9  2015 RPM-GPG-KEY-CentOS-Testing-7
-rw-r--r-- 1 nobody 500  354 Nov 25 17:21 .treeinfo

These rsyncs are all going fine.

rsync -rlptDvSHP --delete  --exclude config.repo --exclude "local*"
--exclude "isos"
anonym...@mirror.ams1.nl.leaseweb.net::centos/7/os/x86_64/Packages/
/localpath/CentOS7-x86_64/Packages/
rsync -rlptDvSHP --delete  --exclude config.repo --exclude "local*"
--exclude "isos"
anonym...@mirror.ams1.nl.leaseweb.net::centos/7/os/x86_64/repodata/
/localpath/CentOS7-x86_64/repodata/
rsync -rlptDvSHP --delete  --exclude config.repo --exclude "local*"
--exclude "isos"
anonym...@mirror.ams1.nl.leaseweb.net::centos/7/os/x86_64/LiveOS/
/localpath/CentOS7-x86_64/LiveOS/

Except when I try to rsync the file CentOS_BuildTag then everything
stalls. Leaving such files
-rw--- 1 500 500 0 Dec  9 14:26 .CentOS_BuildTag.2igwc5
-rw--- 1 500 500 0 Dec  9 14:28 .CentOS_BuildTag.tkiwc5


So something is failing on the write, it seems.  These are the temporary 
files made by rsync, and they're empty, so the initial write seems to 
have failed.



I can resolf this by doing a wget and moving the file to the location
wget
'http://mirror.ams1.nl.leaseweb.net/centos/7/os/x86_64/CentOS_BuildTag'
mv CentOS_BuildTag /localpath/CentOS7-x86_64/

I had also problems with .discinfo and when I ls this directory on
cephfs mount it takes a long time to produce output.

When I do the full rsync to the cephfs mount it completes without
errors, when I then later do the sync on the nfs mount it completes also
(nothing being copied)


This confirms that it's not metadata related, as this second successful 
rsync is purely metadata.



Anybody know what I should do to resolv this? Is this a typical ganesha
issue or is this cephfs corruption, that make ganesha stall?


Writes in Ganesha are pretty much passthrough, modulo some metadata 
tracking.  This means that a write hang is likely to be somewhere 
between Ganesha and CephFS.  However, this is a single, small file, so I 
don't see how it could hang, especially when wget can copy the file 
correctly.  Maybe there's something about the structure of the file? 
Does it have holes in it, for example?


Also, can you send the mount options for the NFS mount?

Daniel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Secure way to wipe a Ceph cluster

[Daniel Gryniewicz]

On 07/27/2018 03:03 AM, Robert Sander wrote:

Hi,

On 27.07.2018 09:00, Christopher Kunz wrote:


as part of deprovisioning customers, we regularly have the task of
wiping their Ceph clusters. Is there a certifiable, GDPR compliant way
to do so without physically shredding the disks?


In the past I have used DBAN from https://dban.org/, but they seem to
follow a more commercial business model now.



Encrypt the drives, and shred the keys?

Daniel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS MDS stuck (failed to rdlock when getattr / lookup)

[Daniel Gryniewicz]

On 05/01/2018 01:43 PM, Oliver Freyermuth wrote:

Hi all,

Am 17.04.2018 um 19:38 schrieb Oliver Freyermuth:

Am 17.04.2018 um 19:35 schrieb Daniel Gryniewicz:

On 04/17/2018 11:40 AM, Oliver Freyermuth wrote:

Am 17.04.2018 um 17:34 schrieb Paul Emmerich:



[...]


  We are right now using the packages from https://eu.ceph.com/nfs-ganesha/ 
<https://eu.ceph.com/nfs-ganesha/> since we would like not to have to build NFS 
Ganesha against Ceph ourselves,
  but would love to just follow upstream to save the maintenance burden. 
Are you building packages yourself, or using a repo maintained upstream?


We are building it ourselves. We plan to soon publish our own repository for it.


This is certainly interesting for us!
Ideally, we would love something with a similar maintenance-promise as the 
eu.ceph.com repositories, always in sync with upstream's ceph releases.
I still hope NFS Ganesha will play a larger role in Mimic and later releases 
any maybe become a more integral part of the Ceph ecosystem.

This is also the main problem preventing it from building it ourselves - if we 
had to do it, we would have to build Ceph and NFS Ganesha, e.g. on the Open 
Build Service,
and closely monitor upstream ourselves.

Many thanks and looking forward towards the publishing of your repo,
 Oliver


2.6 packages are now up on download.ceph.com, thanks to Ali.


Many thanks to Ali also from my side!
We'll schedule the upgrade within the next days and observe closely whether the 
problem reappears.

Many thanks and all the best,
Oliver


Finally, after watching the situation for ~2 weeks after the update, I can say:
Upgrading NFS Ganesha to 2.6 appears to have solved the issue.
I did not observe any further lockup like this anymore, even though our users 
tried their best to re-trigger the issue!
So it seems this has been solved for good by the update :-).

So many thanks and all the best,
Oliver


Thanks, that's good to know.

Daniel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS MDS stuck (failed to rdlock when getattr / lookup)

[Daniel Gryniewicz]

On 04/17/2018 11:40 AM, Oliver Freyermuth wrote:

Am 17.04.2018 um 17:34 schrieb Paul Emmerich:



[...]


 We are right now using the packages from https://eu.ceph.com/nfs-ganesha/ 
 since we would like not to have to build NFS 
Ganesha against Ceph ourselves,
 but would love to just follow upstream to save the maintenance burden. Are 
you building packages yourself, or using a repo maintained upstream?


We are building it ourselves. We plan to soon publish our own repository for it.


This is certainly interesting for us!
Ideally, we would love something with a similar maintenance-promise as the 
eu.ceph.com repositories, always in sync with upstream's ceph releases.
I still hope NFS Ganesha will play a larger role in Mimic and later releases 
any maybe become a more integral part of the Ceph ecosystem.

This is also the main problem preventing it from building it ourselves - if we 
had to do it, we would have to build Ceph and NFS Ganesha, e.g. on the Open 
Build Service,
and closely monitor upstream ourselves.

Many thanks and looking forward towards the publishing of your repo,
Oliver


2.6 packages are now up on download.ceph.com, thanks to Ali.

Daniel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cephfs fsal + nfs-ganesha + el7/centos7

[Daniel Gryniewicz]

To my knowledge, no one has done any work on ganesha + ceph and selinux. 
 Fedora (and RHEL) includes config in it's selinux package for ganesha 
+ gluster, but I'm sure there's missing bits for ceph.


Daniel

On 02/17/2018 03:15 PM, Oliver Freyermuth wrote:

Hi together,

many thanks for the RPMs provided at:
   http://download.ceph.com/nfs-ganesha/
They are very much appreciated!


Since the statement was that they will also be maintained in the future, and 
NFS Ganesha seems an important project for the future of Ceph,
let me do the first "packaging" bug report.

It seems that the current packages do not play so well with SELinux. I'm 
currently using an SELinux module with the following allows, found by
iterative use of audit2allow (full ".te" module added at the end of the mail):

allow ganesha_t cyphesis_port_t:tcp_socket name_connect;
allow ganesha_t proc_net_t:file { getattr open read };
allow ganesha_t self:capability dac_override;
allow ganesha_t self:capability setuid;
allow ganesha_t self:capability setgid;

"cyphesis_port_t" is probably needed since its range (tcp: 6767, 6769, 
6780-6799) overlaps with the default ports
recommended for use by OSDs and nfs-ganesha uses libcephfs to talk to them, the 
other caps appear to be needed by nfs-ganesha itself.

With these in place, it seems my setup is working well. Without the "setgid" 
cap, for example, nfs-ganesha just segfaults after the permission denied failure.
Of course, it would be best if they were installed by the package (potentially, 
more restrictive allows are possible with some care).


Please include me in replies, I am not subscribed to the list.

Cheers and all the best,
Oliver



module nfs_ganesha-fix-perms 1.0;

require {
 type proc_net_t;
 type cyphesis_port_t;
 type ganesha_t;
 class capability setuid;
 class capability setgid;
 class capability dac_override;
 class tcp_socket name_connect;
 class file { getattr open read };
}

#= ganesha_t ==
allow ganesha_t cyphesis_port_t:tcp_socket name_connect;
allow ganesha_t proc_net_t:file { getattr open read };
allow ganesha_t self:capability dac_override;
allow ganesha_t self:capability setuid;
allow ganesha_t self:capability setgid;




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] object lifecycle scope

[Daniel Gryniewicz]

Smallest scope is per-bucket.

Daniel

On 02/06/2018 02:24 PM, Robert Stanford wrote:


  Hello Ceph users.  Is object lifecycle (currently expiration) for rgw 
implementable on a per-object basis, or is the smallest scope the bucket?




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] nfs-ganesha rpm build script has not been adapted for this -

[Daniel Gryniewicz]

This was fixed on next (for 2.6, currently in -rc1) but not backported 
to 2.5.


Daniel

On 01/09/2018 12:41 PM, Marc Roos wrote:
  
The script has not been adapted for this - at the end

http://download.ceph.com/nfs-ganesha/rpm-V2.5-stable/luminous/x86_64/

  
nfs-ganesha-rgw-2.5.4-.el7.x86_64.rpm

  ^





-Original Message-
From: Marc Roos
Sent: dinsdag 29 augustus 2017 12:10
To: amare...@redhat.com; Marc Roos; wooer...@gmail.com
Cc: ceph-us...@ceph.com
Subject: RE: [ceph-users] Cephfs fsal + nfs-ganesha + el7/centos7

  
nfs-ganesha-2.5.2-.el7.x86_64.rpm

  ^
Is this correct?

-Original Message-
From: Marc Roos
Sent: dinsdag 29 augustus 2017 11:40
To: amaredia; wooertim
Cc: ceph-users
Subject: Re: [ceph-users] Cephfs fsal + nfs-ganesha + el7/centos7

  
Ali, Very very nice! I was creating the rpm's based on a old rpm source

spec. And it was a hastle to get them to build, and I am not sure if I
even used to correct compile settings.



-Original Message-
From: Ali Maredia [mailto:amare...@redhat.com]
Sent: maandag 28 augustus 2017 22:29
To: TYLin
Cc: Marc Roos; ceph-us...@ceph.com
Subject: Re: [ceph-users] Cephfs fsal + nfs-ganesha + el7/centos7

Marc,

These rpms (and debs) are built with the latest ganesha 2.5 stable
release and the latest luminous release on download.ceph.com:

http://download.ceph.com/nfs-ganesha/

I just put them up late last week, and I will be maintaining them in the
future.

-Ali

- Original Message -

From: "TYLin" 
To: "Marc Roos" 
Cc: ceph-us...@ceph.com
Sent: Sunday, August 20, 2017 11:58:05 PM
Subject: Re: [ceph-users] Cephfs fsal + nfs-ganesha + el7/centos7

You can get rpm from here

https://download.gluster.org/pub/gluster/glusterfs/nfs-ganesha/old/2.3
.0/CentOS/nfs-ganesha.repo

You have to fix the path mismatch error in the repo file manually.


On Aug 20, 2017, at 5:38 AM, Marc Roos 

wrote:




Where can you get the nfs-ganesha-ceph rpm? Is there a repository
that has these?




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Simple RGW Lifecycle processing questions (luminous 12.2.2)

[Daniel Gryniewicz]

On 12/19/2017 07:50 PM, Bryan Banister wrote:

Hi All,

Hey all,

How often does the "lc process" run on RGW buckets in a cluster?

Also is it configurable per bucket or anything?

Tried searching man pages and ceph docs with no luck, so any help 
appreciated!




This is controlled by the rgw_lifecycle_work_time option.  By default, 
it is set to "00:00-06:00", which means LC processing will start at 
midnight local time, and run up until 6 AM local time.  It will continue 
to do work until it runs out of work or runs out of time.


This is a global setting, not a per-bucket setting.

Daniel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph luminous nfs-ganesha-ceph

[Daniel Gryniewicz]

On 12/14/2017 09:46 AM, nigel davies wrote:

Is this nfs-ganesha exporting Cephfs? Yes

Are you using NFS for a Vmware Datastore? Yes

What are you using for the NFS failover? (this is where i could be going 
wrong)


When creating the NFS Datastore i added the two NFS servers ip address in



NFS failover is more complicated than that.  It doesn't natively support 
failover from one machine to another.  Instead, you need to have an 
active/active or active/passive setup on the servers that manages 
virtual IPs, and fails the IP over from one machine to another.  This is 
because NFS, as a protocol, will recover from a failed server if *that 
server* comes back, but not to a second server.  The virtual IP failover 
looks, to the client, like a restart of a single server.


This is done in Ganesha currently with Pacemaker/Corosync to manage the 
failover.  It's in a supported product for Ganesha over Gluster, but 
not, as far as I know, for Ganesha over Ceph.  It has been done by 
community members, however.


The Gluster docs for this are here:
http://docs.gluster.org/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/

In the future (Ganesha 2.6 / Ceph Mimic) this should be better supported 
over Ceph.


Daniel




On Thu, Dec 14, 2017 at 2:29 PM, David C > wrote:


Is this nfs-ganesha exporting Cephfs?
Are you using NFS for a Vmware Datastore?
What are you using for the NFS failover?

We need more info but this does sound like a vmware/nfs question
rather than specifically ceph/nfs-ganesha

On Thu, Dec 14, 2017 at 1:47 PM, nigel davies > wrote:

Hay all

i am in the process or trying to set up and VMware storage
environment
I been reading and found that Iscsi (on jewel release) can cause
issues and the datastore can drop out.

I been looking at using  nfs-ganesha with my ceph platform, it
all looked good until i looked at failover to our 2nd nfs serve.
I believe i set up the Server right. gave the two IP address of
the NFS servers.

when i shutdown the live NFS server, the datastore becomes
"inactive" even after i bring the NFS server backup, it stil
shows as inactive.

any advise on this i would be grateful incase i am missing
something.
i am using NFS 4.1 as i been advised it will support the fail over

___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com






___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ganesha NFS

[Daniel Gryniewicz]

Hi.

NFSv3 is a bit different than v4.  In the case of v3, you need to mount 
the full path, rather than the Pseudo path.  This can cause problems for 
cephfs, because you probably exported / from cephfs.


A good solution to this is to set

mount_path_pseudo = true;

in the NFS_CORE_PARAM block.  This will cause v3 to mount by pseudo 
path, and therefore take the same mount arguments as v4 does.


Daniel

On 11/09/2017 06:05 AM, jorpilo wrote:

Hi!
I would like to export my cephfs using Ganesha NFS to export a NFSv3 o NFSv4

I am a little lost while doing it. I managed to make it working with 
NFSv4 but I can't make it work with NFSv3 as the server refuses the 
connection.


Has anyone managed to do it?


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Kraken rgw lifeycle processing nightly crash

[Daniel Gryniewicz]

On 07/20/2017 04:48 PM, Ben Hines wrote:
Still having this RGWLC crash once a day or so. I do plan to update to 
Luminous as soon as that is final, but it's possible this issue will 
still occur, so i was hoping one of the devs could take a look at it.


My original suspicion was that it happens when lifecycle processing at 
the same time that the morning log rotation occurs, but i am not certain 
about that, so perhaps the bug title should be updated to remove that 
conclusion. (i can't edit it)


http://tracker.ceph.com/issues/19956 - no activity for 2 months.

Stack with symbols:

#0 0x7f6a6cb1723b in raise () from /lib64/libpthread.so.0
#1  0x7f6a778b9e95 in 
reraise_fatal (signum=11) at 
/usr/src/debug/ceph-11.2.0/src/global/signal_handler.cc:72
#2  handle_fatal_signal (signum=11) at 
/usr/src/debug/ceph-11.2.0/src/global/signal_handler.cc:134

#3  
#4  RGWGC::add_chain 
(this=this@entry=0x0, op=..., chain=..., 
tag="default.68996150.61684839") at 
/usr/src/debug/ceph-11.2.0/src/rgw/rgw_gc.cc:58
#5  0x7f6a77801e3f in 
RGWGC::send_chain (this=0x0, chain=..., tag="default.68996150.61684839", 
sync=sync@entry=false)


Here, this (the RGWGC, or store->gc) is NULL, so that's the problem.  I 
have no idea how the store isn't initialized, though.



at /usr/src/debug/ceph-11.2.0/src/rgw/rgw_gc.cc:64
#6  0x7f6a776c0a29 in 
RGWRados::Object::complete_atomic_modification (this=0x7f69cc8578d0) at 
/usr/src/debug/ceph-11.2.0/src/rgw/rgw_rados.cc:7870
#7  0x7f6a777102a0 in 
RGWRados::Object::Delete::delete_obj (this=this@entry=0x7f69cc857840) at 
/usr/src/debug/ceph-11.2.0/src/rgw/rgw_rados.cc:8295
#8  0x7f6a77710ce8 in 
RGWRados::delete_obj (this=, obj_ctx=..., 
bucket_info=..., obj=..., versioning_status=0, bilog_flags=,

expiration_time=...) at /usr/src/debug/ceph-11.2.0/src/rgw/rgw_rados.cc:8330
#9  0x7f6a77607ced in 
rgw_remove_object (store=0x7f6a810fe000, bucket_info=..., bucket=..., 
key=...) at /usr/src/debug/ceph-11.2.0/src/rgw/rgw_bucket.cc:519
#10  0x7f6a7780c971 in 
RGWLC::bucket_lc_process (this=this@entry=0x7f6a81959c00, 
shard_id=":globalcache307:default.42048218.11")

at /usr/src/debug/ceph-11.2.0/src/rgw/rgw_lc.cc:283
#11  0x7f6a7780d928 in 
RGWLC::process (this=this@entry=0x7f6a81959c00, index=, 
max_lock_secs=max_lock_secs@entry=60)

at /usr/src/debug/ceph-11.2.0/src/rgw/rgw_lc.cc:482
#12  0x7f6a7780ddc1 in 
RGWLC::process (this=0x7f6a81959c00) at 
/usr/src/debug/ceph-11.2.0/src/rgw/rgw_lc.cc:412
#13  0x7f6a7780e033 in 
RGWLC::LCWorker::entry (this=0x7f6a81a820d0) at 
/usr/src/debug/ceph-11.2.0/src/rgw/rgw_lc.cc:51
#14  0x7f6a6cb0fdc5 in 
start_thread () from /lib64/libpthread.so.0
#15  0x7f6a6b37073d in clone () 
from /lib64/libc.so.6




Daniel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Problems getting nfs-ganesha with cephfs backend to work.

[Daniel Gryniewicz]

On 07/19/2017 05:27 AM, Micha Krause wrote:

Hi,

Ganesha version 2.5.0.1 from the nfs-ganesha repo hosted on 
download.ceph.com 


I didn't know about that repo, and compiled ganesha myself. The 
developers in the #ganesha IRC channel pointed me to

the libcephfs version.
After recompiling ganesha with a kraken libcephfs instead of a jewel 
version both errors went away.


I'm sure using a compiled Version from the repo you mention would have 
worked out of the box.


Micha Krause



These packages aren't quite ready for use yet, the packaging work is 
still underway.  CCing Ali, who's doing the work.


Daniel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW lifecycle not expiring objects

[Daniel Gryniewicz]

On 06/28/2017 02:30 PM, Graham Allan wrote:

That seems to be it! I couldn't see a way to specify the auth version
with aws cli (is there a way?). However it did work with s3cmd and v2 auth:

% s3cmd --signature-v2 setlifecycle lifecycle.xml s3://testgta
s3://testgta/: Lifecycle Policy updated


Good stuff.



(I believe that with Kraken, this threw an error and failed to set the
policy, but I'm not certain at this point... besides which radosgw
didn't then have access to the default.rgw.lc pool, which may have
caused further issues)

No way to read the lifecycle policy back with s3cmd, so:


I submitted a patch a while ago to add getlifecycle to s3cmd, and it was 
accepted, but I don't know about releases or distro packaging.  It will 
be there eventually.




% aws --endpoint-url https://xxx.xxx.xxx.xxx s3api \
get-bucket-lifecycle-configuration --bucket=testgta
{
"Rules": [
{
"Status": "Enabled",
"Prefix": "",
"Expiration": {
"Days": 1
},
"ID": "test"
}
]
}

and looks encouraging at the server side:

#  radosgw-admin lc list
[
{
"bucket": ":gta:default.6985397.1",
"status": "UNINITIAL"
},
{
"bucket": ":testgta:default.6790451.1",
"status": "UNINITIAL"
}
]

then:
#  radosgw-admin lc process

and all the (very old) objects disappeared from the test bucket.


Good to know.

Daniel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW lifecycle not expiring objects

[Daniel Gryniewicz]

Kraken has lifecycle, Jewel does not.

Daniel

On 06/04/2017 07:16 PM, ceph.nov...@habmalnefrage.de wrote:


grrr... sorry && and again as text :|


Gesendet: Montag, 05. Juni 2017 um 01:12 Uhr
Von: ceph.nov...@habmalnefrage.de
An: "Yehuda Sadeh-Weinraub" 
Cc: "ceph-users@lists.ceph.com" , 
ceph-de...@vger.kernel.org
Betreff: Re: [ceph-users] RGW lifecycle not expiring objects



Hi (again) Yehuda.

Looping in ceph-devel...

Could it be that lifecycle is still not implemented neither in Jewel nor in 
Kraken, even if release notes and other places say so?

https://www.spinics.net/lists/ceph-devel/msg34492.html
https://github.com/ceph/ceph-ci/commit/7d48f62f5c86913d8f00b44d46a04a52d338907c
https://github.com/ceph/ceph-ci/commit/9162bd29594d34429a09562ed60a32a0703940ea

Thanks & regards
 Anton


Gesendet: Sonntag, 04. Juni 2017 um 21:34 Uhr
Von: ceph.nov...@habmalnefrage.de
An: "Yehuda Sadeh-Weinraub" 
Cc: "ceph-users@lists.ceph.com" 
Betreff: Re: [ceph-users] RGW lifecycle not expiring objects
Hi Yahuda.

Well, here we go: 
http://tracker.ceph.com/issues/20177[http://tracker.ceph.com/issues/20177]

As it's my first one, hope it's ok as it is...

Thanks & regards
Anton


Gesendet: Samstag, 03. Juni 2017 um 00:14 Uhr
Von: "Yehuda Sadeh-Weinraub" 
An: ceph.nov...@habmalnefrage.de
Cc: "Graham Allan" , "ceph-users@lists.ceph.com" 

Betreff: Re: [ceph-users] RGW lifecycle not expiring objects
Have you opened a ceph tracker issue, so that we don't lose track of
the problem?

Thanks,
Yehuda

On Fri, Jun 2, 2017 at 3:05 PM,  wrote:

Hi Graham.

We are on Kraken and have the same problem with "lifecycle". Various (other) tools like 
s3cmd or CyberDuck do show the applied "expiration" settings, but objects seem never to 
be purged.

If you should have new findings, hints,... PLEASE share/let me know.

Thanks a lot!
Anton


Gesendet: Freitag, 19. Mai 2017 um 22:44 Uhr
Von: "Graham Allan" 
An: ceph-users@lists.ceph.com
Betreff: [ceph-users] RGW lifecycle not expiring objects
I've been having a hard time getting the s3 object lifecycle to do
anything here. I was able to set a lifecycle on a test bucket. As others
also seem to have found, I do get an EACCES error on setting the
lifecycle, but it does however get stored:


% aws --endpoint-url 
https://xxx.xxx.xxx.xxx[https://xxx.xxx.xxx.xxx][https://xxx.xxx.xxx.xxx[https://xxx.xxx.xxx.xxx]]
 s3api get-bucket-lifecycle-configuration --bucket=testgta
{
"Rules": [
{
"Status": "Enabled",
"Prefix": "",
"Expiration": {
"Days": 3
},
"ID": "test"
}
]
}


but many days later I have yet to see any object actually get expired.
There are some hints in the rgw log that the expiry thread does run
periodically:


2017-05-19 03:49:03.281347 7f74f1134700 2 
RGWDataChangesLog::ChangesRenewThread: start
2017-05-19 03:49:16.356022 7f74ef931700 2 object expiration: start
2017-05-19 03:49:16.356036 7f74ef931700 20 proceeding shard = 
obj_delete_at_hint.00
2017-05-19 03:49:16.359785 7f74ef931700 20 proceeding shard = 
obj_delete_at_hint.01
2017-05-19 03:49:16.364667 7f74ef931700 20 proceeding shard = 
obj_delete_at_hint.02
2017-05-19 03:49:16.369636 7f74ef931700 20 proceeding shard = 
obj_delete_at_hint.03

...

2017-05-19 03:49:16.803270 7f74ef931700 20 proceeding shard = 
obj_delete_at_hint.000126
2017-05-19 03:49:16.806423 7f74ef931700 2 object expiration: stop


"radosgw-admin lc process" gives me no output unless I enable debug, then:


]# radosgw-admin lc process
2017-05-19 15:28:46.383049 7fedb9ffb700 2 
RGWDataChangesLog::ChangesRenewThread: start
2017-05-19 15:28:46.421806 7feddc240c80 10 Cannot find current period zone 
using local zone
2017-05-19 15:28:46.453431 7feddc240c80 2 all 8 watchers are set, enabling cache
2017-05-19 15:28:46.614991 7feddc240c80 2 removed watcher, disabling cache


"radosgw-admin lc list" seems to return "empty" output:


# radosgw-admin lc list
[]


Is there anything obvious that I might be missing?

Graham
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users mailing list
ceph-users@lists.ceph.com

Re: [ceph-users] RGW authentication fail with AWS S3 v4

[Daniel Gryniewicz]

It looks like, as it's now coded, the 15 minute time limit is hard 
coded.  It checks that X-Amz-Expires is not exceeded, and then 
unconditionally checks that the request time is within 15 minutes of now.


Daniel

On 02/03/2017 04:06 AM, Khang Nguyễn Nhật wrote:

Dear Wido,

I have used X-Amz-Expires=86400 in url but it doesn't work

2017-02-03 16:00 GMT+07:00 Wido den Hollander >:


> Op 3 februari 2017 om 9:52 schreef Khang Nguyễn Nhật
>:
>
>
> Hi all,
> I'm using Ceph Object Gateway with S3 API 
(ceph-radosgw-10.2.5-0.el7.x86_64
> on CentOS Linux release 7.3.1611) and  I use generate_presigned_url method
> of boto3 to create rgw url. This url working fine in period of 15 minutes,
> after 15 minutes I recived *RequestTimeTooSkewed* error. My
radosgw use
> Asia/Ho_Chi_Minh timezone and running ntp service. Here is url and rgw 
log:
>

That is normal. The time is part of the signature. You have to
generate a new signature after 15 minutes.

Normal behavior.

Wido

> - URL:
> 
http://rgw.xxx.vn/bucket/key.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256=86400=7AHTO4E1JBZ1VG1U96F1%2F20170203%2F%2Fs3%2Faws4_request=host=20170203T081233Z=682be59232443fee58bc4744f656c533da8ddd828e36b739b332736fa22bef51


>
> - RGW LOG:
> // //
> NOTICE: request time skew too big.
> now_req = 1486109553 now = 1486110512; now -
> RGW_AUTH_GRACE_MINS=1486109612; now + RGW_AUTH_GRACE_MINS=1486111412
> failed to authorize request
> handler->ERRORHANDLER: err_no=-2012 new_err_no=-2012
> // //
>
> Someone can help me reslove this problem ? Thank
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com





___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How exactly does rgw work?

[Daniel Gryniewicz]

Yes, this is common practice.

Daniel

On 12/22/2016 02:34 PM, Gerald Spencer wrote:

Wonderful, just as I expected. Do folks normally have several RGW
running on individual machines with a load balancer at larger scales?

On Wed, Dec 21, 2016 at 8:22 AM, LOPEZ Jean-Charles > wrote:

Hi Gerald,

for the s3 and swift case, the clients are not accessing the ceph
cluster. They are s3 and swift clients and only discuss with the RGW
over HTTP. The RGW is the ceph client that does all the interaction
with the ceph cluster.

Best
JC


On Dec 21, 2016, at 07:27, Gerald Spencer > wrote:

I was under the impression that when a client talks to the
cluster, it grabs the osd map and computes the crush algorithm to
determine where it stores the object. Does the rgw server do this
for clients? If I had 12 clients all talking through one gateway,
would that server have to pass all of the objects from the clients
to the cluster?


And 48 osd nodes, each with 12 x 6TB drives and a PCIe write
journal. That would be 576 osds in the cluster, with about 3.4PB
raw...


On Tue, Dec 20, 2016 at 1:12 AM Wido den Hollander > wrote:



> Op 20 december 2016 om 3:24 schreef Gerald Spencer
>:

>

>

> Hello all,

>

> We're currently waiting on a delivery of equipment for a
small 50TB proof

> of concept cluster, and I've been lurking/learning a ton
from you. Thanks

> for how active everyone is.

>

> Question(s):

> How does the raids gateway work exactly?



The RGW doesn't do any RAID. It chunks up larger objects into
smaller RADOS chunks. The first chunk is always 512k (IIRC)
and then it chunks up into 4MB RADOS objects.



> Does it introduce a single point of failure?



It does if you deploy only one RGW. Always deploy multiple
with loadbalancing in front.



> Does all of the traffic go through the host running the rgw
server?



Yes it does.



>

> I just don't fully understand that side of things. As for
architecture our

> poc will have:

> - 1 monitor

> - 4 OSDs with 12 x 6TB drives, 1 x 800 PCIe journal

>



Underscaled machines, go for less disks per machine but more
machines. More smaller machines works a lot better with Ceph
then a few big machines.



> I'd all goes as planned, this will scale up to:

> - 3 monitors



Always run with 3 MONs. Otherwise it is a serious SPOF.



> - 48 osds

>

> This should give us enough storage (~1.2PB) wth enough
throughput to handle

> the data requirements of our machines to saturate our 100Gb
link...

>



That won't happen with just 4 machines. Replica 3x taken into
account is well. You will need a lot more machines to get the
100Gb link fully utilized.



Wido



>

>

>

>

> Cheers,

> G

> ___

> ceph-users mailing list

> ceph-users@lists.ceph.com 

> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com






___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph and container

[Daniel Gryniewicz]

In addition, Red Hat is shipping a containerized Ceph (all daemons, not 
just mons) as a tech preview in RHCS, and the plan is to support it 
going forward.  We have not seen performance issues related to being 
containerized.  It's based on the ceph-docker and ceph-ansible projects.


Daniel

On 11/15/2016 09:30 AM, John Petrini wrote:

I've had lots of success running monitors in VM's. Never tried the
container route but there is a ceph-docker
project https://github.com/ceph/ceph-docker if you want to give it a
shot. I don't know how highly recommended that it though, I've got no
personal experience with it.

No matter what you want to make sure you don't have a single point of
failure. There's not much point it having three monitors if they are all
going to run in containers/VM's on the same host.

___

John Petrini

NOC Systems Administrator   //   *CoreDial, LLC*   //   coredial.com
   //   Twitter
   LinkedIn
   Google Plus
   Blog

Hillcrest I, 751 Arbor Way, Suite 150, Blue Bell PA, 19422
*P: *215.297.4400 x232
//   *F: *215.297.4401   //   *E: *jpetr...@coredial.com


Exceptional people. Proven Processes. Innovative Technology. Discover
CoreDial - watch our video


The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission,  dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you
received this in error, please contact the sender and delete the
material from any computer.



On Tue, Nov 15, 2016 at 9:25 AM, Matteo Dacrema > wrote:

Hi,

does anyone ever tried to run ceph monitors in containers?
Could it lead to performance issues?
Can I run monitor containers on the OSD nodes?

I don’t want to buy 3 dedicated servers. Is there any other solution?

Thanks
Best regards

Matteo Dacrema


___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com





___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw - http status 400 while creating a bucket

[Daniel Gryniewicz]

Your RGW doesn't think it's the master, and cannot connect to the 
master, thus the create fails.


Daniel

On 11/08/2016 06:36 PM, Andrei Mikhailovsky wrote:

Hello

I am having issues with creating buckets in radosgw. It started with an
upgrade to version 10.2.x

When I am creating a bucket I get the following error on the client side:


boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
InvalidArgumentmy-new-bucket-31337tx2-0058225bae-994d148-default994d148-default-default


The radosgw logs are (redacted):

###

2016-11-08 23:11:42.876862 7f026d953700 20 enqueued request
req=0x7f02ba07b0e0
2016-11-08 23:11:42.876892 7f026d953700 20 RGWWQ:
2016-11-08 23:11:42.876897 7f026d953700 20 req: 0x7f02ba07b0e0
2016-11-08 23:11:42.876912 7f026d953700 10 allocated request
req=0x7f02ba07b140
2016-11-08 23:11:42.876975 7f026b94f700 20 dequeued request
req=0x7f02ba07b0e0
2016-11-08 23:11:42.876987 7f026b94f700 20 RGWWQ: empty
2016-11-08 23:11:42.877050 7f026b94f700 20 CONTENT_LENGTH=0
2016-11-08 23:11:42.877060 7f026b94f700 20 CONTEXT_DOCUMENT_ROOT=/var/www
2016-11-08 23:11:42.877062 7f026b94f700 20 CONTEXT_PREFIX=
2016-11-08 23:11:42.877063 7f026b94f700 20 DOCUMENT_ROOT=/var/www
2016-11-08 23:11:42.877081 7f026b94f700 20 FCGI_ROLE=RESPONDER
2016-11-08 23:11:42.877083 7f026b94f700 20 GATEWAY_INTERFACE=CGI/1.1
2016-11-08 23:11:42.877084 7f026b94f700 20 HTTP_ACCEPT_ENCODING=identity
2016-11-08 23:11:42.877086 7f026b94f700 20 HTTP_AUTHORIZATION=AWS
XXXEDITEDXX:EDITEDXXXeWyiacaN26GcME
2016-11-08 23:11:42.877087 7f026b94f700 20 HTTP_DATE=Tue, 08 Nov 2016
23:11:37 GMT
2016-11-08 23:11:42.877088 7f026b94f700 20
HTTP_HOST=s3service.editedname.com
2016-11-08 23:11:42.877089 7f026b94f700 20 HTTP_USER_AGENT=Boto/2.38.0
Python/2.7.12 Linux/4.8.4-040804-generic
2016-11-08 23:11:42.877090 7f026b94f700 20 HTTPS=on
2016-11-08 23:11:42.877092 7f026b94f700 20
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2016-11-08 23:11:42.877093 7f026b94f700 20 proxy-nokeepalive=1
2016-11-08 23:11:42.877094 7f026b94f700 20 QUERY_STRING=
2016-11-08 23:11:42.877095 7f026b94f700 20 REMOTE_ADDR=192.168.169.91
2016-11-08 23:11:42.877096 7f026b94f700 20 REMOTE_PORT=45404
2016-11-08 23:11:42.877097 7f026b94f700 20 REQUEST_METHOD=PUT
2016-11-08 23:11:42.877098 7f026b94f700 20 REQUEST_SCHEME=https
2016-11-08 23:11:42.877099 7f026b94f700 20 REQUEST_URI=/my-new-bucket-31337/
2016-11-08 23:11:42.877104 7f026b94f700 20
SCRIPT_FILENAME=proxy:fcgi://localhost:9000/my-new-bucket-31337/
2016-11-08 23:11:42.877105 7f026b94f700 20 SCRIPT_NAME=/my-new-bucket-31337/
2016-11-08 23:11:42.877107 7f026b94f700 20
SCRIPT_URI=https://s3service.editedname.com/my-new-bucket-31337/
2016-11-08 23:11:42.877108 7f026b94f700 20 SCRIPT_URL=/my-new-bucket-31337/
2016-11-08 23:11:42.877109 7f026b94f700 20 SERVER_ADDR=192.168.169.201
2016-11-08 23:11:42.877110 7f026b94f700 20
SERVER_ADMIN=and...@editedname.com
2016-11-08 23:11:42.877111 7f026b94f700 20
SERVER_NAME=s3service.editedname.com
2016-11-08 23:11:42.877112 7f026b94f700 20 SERVER_PORT=443
2016-11-08 23:11:42.877113 7f026b94f700 20 SERVER_PROTOCOL=HTTP/1.1
2016-11-08 23:11:42.877114 7f026b94f700 20 SERVER_SIGNATURE=
2016-11-08 23:11:42.877115 7f026b94f700 20 SERVER_SOFTWARE=Apache/2.4.18
(Ubuntu)
2016-11-08 23:11:42.877116 7f026b94f700 20
SSL_TLS_SNI=s3service.editedname.com
2016-11-08 23:11:42.877119 7f026b94f700  1 == starting new request
req=0x7f02ba07b0e0 =
2016-11-08 23:11:42.877155 7f026b94f700  2 req 2:0.35::PUT
/my-new-bucket-31337/::initializing for trans_id =
tx2-0058225bae-994d148-default
2016-11-08 23:11:42.877175 7f026b94f700 10 rgw api priority: s3=5
s3website=4
2016-11-08 23:11:42.877179 7f026b94f700 10 host=s3service.editedname.com
2016-11-08 23:11:42.877199 7f026b94f700 20 subdomain=
domain=s3service.editedname.com in_hosted_domain=1
in_hosted_domain_s3website=0
2016-11-08 23:11:42.877203 7f026b94f700 20 final domain/bucket
subdomain= domain=s3service.editedname.com in_hosted_domain=1
in_hosted_domain_s3website=0 s->info.domain=s3service.editedname.com
s->info.request_uri=/my-new-bucket-31337/
2016-11-08 23:11:42.877277 7f026b94f700 20 get_handler
handler=25RGWHandler_REST_Bucket_S3
2016-11-08 23:11:42.877286 7f026b94f700 10
handler=25RGWHandler_REST_Bucket_S3
2016-11-08 23:11:42.877291 7f026b94f700  2 req 2:0.000172:s3:PUT
/my-new-bucket-31337/::getting op 1
2016-11-08 23:11:42.877326 7f026b94f700 10 op=27RGWCreateBucket_ObjStore_S3
2016-11-08 23:11:42.877334 7f026b94f700  2 req 2:0.000215:s3:PUT
/my-new-bucket-31337/:create_bucket:authorizing
2016-11-08 23:11:42.877386 7f026b94f700 20 get_system_obj_state:
rctx=0x7f026b94b7c0 obj=.users:XXXEDITEDXX state=0x7f02b70cdfe8
s->prefetch_data=0
2016-11-08 23:11:42.877403 7f026b94f700 10 cache get:
name=.users+XXXEDITEDXX : miss
2016-11-08 23:11:42.877516 7f026b94f700  1 --
192.168.168.201:0/3703211654 --> 

Re: [ceph-users] UID reset to root after chgrp on CephFS Ganesha export

[Daniel Gryniewicz]

On 08/31/2016 02:15 PM, Wido den Hollander wrote:



Op 31 augustus 2016 om 15:28 schreef Daniel Gryniewicz <d...@redhat.com>:


I believe this is a Ganesha bug, as discussed on the Ganesha list.



Ah, thanks. Do you maybe have a link or subject so I can chime in?

Wido


https://sourceforge.net/p/nfs-ganesha/mailman/message/35321818/

Daniel




Daniel

On 08/31/2016 06:55 AM, Wido den Hollander wrote:



Op 31 augustus 2016 om 12:42 schreef John Spray <jsp...@redhat.com>:


On Wed, Aug 31, 2016 at 11:23 AM, Wido den Hollander <w...@42on.com> wrote:

Hi,

I have a CephFS filesystem which is re-exported through NFS Ganesha (v2.3.0) 
with Ceph 10.2.2

The export works fine, but when calling a chgrp on a file the UID is set to 
root.

Example list of commands:

$ chown www-data:www-data myfile

That works, file is now owned by www-data/www-data

$ chgrp nogroup myfile


Does the nogroup group have GID -1 by any chance?  There is a bug in
the userspace client where it doesn't handle negative gid/uids
properly at all, which Greg is working on at the moment
(http://tracker.ceph.com/issues/16367)



No, it doesn't. nogroup is just an example here. This happens with any group.

The UID is always set back to 0.

Wido


John


That files, the UID is now set to 0 (root) and the group hasn't changed.

I tracked this down to being a Ganesha problem in combination with CephFS. 
Running these commands on either a kernel mounted CephFS or via FUSE I don't 
see this problem.

The Ganesha configuration:

NFSv4
{
IdmapConf = /etc/idmapd.conf;
}

EXPORT
{
Export_ID = 1;
Path = "/";
Pseudo = "/";
Access_Type = RW;
Protocols = "4";
Squash = no_root_squash;
Transports = TCP;
SecType = sys;

FSAL {
Name = CEPH;
}
}

Has anybody seen this before?

Wido
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] UID reset to root after chgrp on CephFS Ganesha export

[Daniel Gryniewicz]

I believe this is a Ganesha bug, as discussed on the Ganesha list.

Daniel

On 08/31/2016 06:55 AM, Wido den Hollander wrote:



Op 31 augustus 2016 om 12:42 schreef John Spray :


On Wed, Aug 31, 2016 at 11:23 AM, Wido den Hollander  wrote:

Hi,

I have a CephFS filesystem which is re-exported through NFS Ganesha (v2.3.0) 
with Ceph 10.2.2

The export works fine, but when calling a chgrp on a file the UID is set to 
root.

Example list of commands:

$ chown www-data:www-data myfile

That works, file is now owned by www-data/www-data

$ chgrp nogroup myfile


Does the nogroup group have GID -1 by any chance?  There is a bug in
the userspace client where it doesn't handle negative gid/uids
properly at all, which Greg is working on at the moment
(http://tracker.ceph.com/issues/16367)



No, it doesn't. nogroup is just an example here. This happens with any group.

The UID is always set back to 0.

Wido


John


That files, the UID is now set to 0 (root) and the group hasn't changed.

I tracked this down to being a Ganesha problem in combination with CephFS. 
Running these commands on either a kernel mounted CephFS or via FUSE I don't 
see this problem.

The Ganesha configuration:

NFSv4
{
IdmapConf = /etc/idmapd.conf;
}

EXPORT
{
Export_ID = 1;
Path = "/";
Pseudo = "/";
Access_Type = RW;
Protocols = "4";
Squash = no_root_squash;
Transports = TCP;
SecType = sys;

FSAL {
Name = CEPH;
}
}

Has anybody seen this before?

Wido
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Radosgw admin ops API command question

[Daniel Gryniewicz]

On 07/21/2016 05:04 AM, Horace wrote:

Hello all,

I've a question regarding Ceph Radosgw Admin ops API would like to ask your 
help. I couldn't find the API equivalent of the follow CLI command to look up 
the total bytes of a user. Your help will be appreciated, thanks in advance!

radosgw-admin user stats --uid=user1

http://docs.ceph.com/docs/master/radosgw/adminops/


Regards,
Horace Ng


It looks like the "Get User Info" API call can return stats by passing a 
boolean argument stats=true.  This is not in the documentation, but is 
in the code.


Daniel

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osd inside LXC

[Daniel Gryniewicz]

This is fairly standard for container deployment: one app per container 
instance.  This is how we're deploying docker in our upstream 
ceph-docker / ceph-ansible as well.


Daniel

On 07/13/2016 08:41 PM, Łukasz Jagiełło wrote:

Hi,

Just wonder why you want each OSD inside separate LXC container? Just to
pin them to specific cpus?

On Tue, Jul 12, 2016 at 6:33 AM, Guillaume Comte
> wrote:

Hi,

I am currently defining a storage architecture based on ceph, and i
wish to know if i don't misunderstood some stuffs.

So, i plan to deploy for each HDD of each servers as much as OSD as
free harddrive, each OSD will be inside a LXC container.

Then, i wish to turn the server itself as a rbd client for objects
created in the pools, i wish also to have a SSD to activate caching
(and also store osd logs as well)

The idea behind is to create CRUSH rules which will maintain a set
of object within a couple of servers connected to the same pair of
switch in order to have the best proximity between where i store the
object and where i use them (i don't bother having a very high
insurance to not loose data if my whole rack powerdown)

Am i already on the wrong track ? Is there a way to guaranty
proximity of data with ceph whitout making twisted configuration as
i am ready to do ?

Thks in advance,

Regards
--
*Guillaume Comte*
06 25 85 02 02  | guillaume.co...@blade-group.com

90 avenue des Ternes, 75 017 Paris


___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




--
Łukasz Jagiełło
lukaszjagielloorg


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OSPF to the host

[Daniel Gryniewicz]

On 07/11/2016 08:23 AM, Saverio Proto wrote:

I'm looking at the Dell S-ON switches which we can get in a Cumulus
version. Any pro's and con's of using Cumulus vs old school switch OS's you
may have come across?


Nothing to declare here. Once configured properly the hardware works
as expected. I never used Dell, I used switches from Quanta.

Saverio


I've had good experiences with Dell switches in the past, including routing.

Daniel

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph/daemon mon not working and status exit (1)

[Daniel Gryniewicz]

On 07/07/2016 08:06 PM, Rahul Talari wrote:

I am trying to use Ceph in Docker. I have built the ceph/base and
ceph/daemon DockeFiles. I am trying to deploy a Ceph monitor according
to the instructions given in the tutorial but when I execute the command
without KV store and type:

sudo docker ps

I am not able to keep the monitor up. What mistakes am I performing with
doing so? Is there something I should do to get it up and running
continuously without failing?

Thank you



Hi, Rahul.

There are several things you need, most important of which is ceph 
configuration and keys.  These are generated on the host and volume 
mounted into the container.


You can use the "docker logs" to get the output from the failed 
container to see what might be causing the issue.


Daniel

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Running ceph in docker

[Daniel Gryniewicz]

On 06/30/2016 02:05 AM, F21 wrote:

Hey all,

I am interested in running ceph in docker containers. This is extremely
attractive given the recent integration of swarm into the docker engine,
making it really easy to set up a docker cluster.

When running ceph in docker, should monitors, radosgw and OSDs all be on
separate physical nodes? I watched Sebastian's video on setting up ceph
in docker here: https://www.youtube.com/watch?v=FUSTjTBA8f8. In the
video, there were 6 OSDs, with 2 OSDs running on each node.

Is running multiple OSDs on the same node a good idea in production? Has
anyone operated ceph in docker containers in production? Are there any
things I should watch out for?

Cheers,
Francis


It's actually quite common to run multiple OSDs on the same physical 
node, since an OSD currently maps to a single block device.  Depending 
on your load and traffic, it's usually a good idea to run monitors and 
RGWs on separate nodes.


Daniel

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Status of ceph-docker

[Daniel Gryniewicz]

On 05/03/2016 04:17 PM, Vincenzo Pii wrote:

https://github.com/ceph/ceph-docker

Is someone using ceph-docker in production or the project is meant more
for development and experimentation?

Vincenzo Pii| TERALYTICS
*DevOps Engineer
*


I'm not aware of anyone currently using it in production, but it is 
being used as a base for a downstream RHCS containerized release, so 
there will be production containerized Ceph deployed.


Daniel

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph RGW NFS

[Daniel Gryniewicz]

On 02/28/2016 08:36 PM, David Wang wrote:

Hi All,
 How the progress of NFS on RGW? Does it released on Infernalis? The
contents of NFS on RGW is
http://tracker.ceph.com/projects/ceph/wiki/RGW_-_NFS




The FSAL has been integrated into upstream Ganesha 
(https://github.com/nfs-ganesha/nfs-ganesha/tree/next).  It's in testing 
at this point, and so not production ready.  It will be released with 
Jewel (has been merged into master), and may or may not be backported at 
some point.


Daniel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] any recommendation of using EnhanceIO?

[Daniel Gryniewicz]

I did some (non-ceph) work on these, and concluded that bcache was the best 
supported, most stable, and fastest. This was ~1 year ago, to take it with a 
grain of salt, but that's what I would recommend. 

Daniel 


- Original Message -

From: Dominik Zalewski dzalew...@optlink.net 
To: German Anders gand...@despegar.com 
Cc: ceph-users ceph-users@lists.ceph.com 
Sent: Wednesday, July 1, 2015 5:28:10 PM 
Subject: Re: [ceph-users] any recommendation of using EnhanceIO? 

Hi, 

I’ve asked same question last weeks or so (just search the mailing list 
archives for EnhanceIO :) and got some interesting answers. 

Looks like the project is pretty much dead since it was bought out by HGST. 
Even their website has some broken links in regards to EnhanceIO 

I’m keen to try flashcache or bcache (its been in the mainline kernel for some 
time) 

Dominik 




On 1 Jul 2015, at 21:13, German Anders  gand...@despegar.com  wrote: 

Hi cephers, 

Is anyone out there that implement enhanceIO in a production environment? any 
recommendation? any perf output to share with the diff between using it and 
not? 

Thanks in advance, 



German 
___ 
ceph-users mailing list 
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 





___ 
ceph-users mailing list 
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com