Hi, Thank you for this excellent project. Are you still planning on incorporating the code from the Maat paper at some time in the future? As I'm sure you already know, access control lists on the storage objects would be quite useful to store research data.
We are experimenting with a small ceph cluster (3 older Sun V20z servers with 2 SCSI drives each). To give you a brief overview of our setup, we have one MDS/monitoring node and 2 OSDs. The OSDs have 16GB of RAM and the MDS/monitoring node has 12GB. We are using btrfs on a dedicated drive, mounted as /btrfs (each node has this mount). On the OSDs, we also use a partition on the boot disk as /btrfs_journal for the journal and the data is on the second drive, mounted as /btrfs. We only have toy data on the file system, so we can reformat the ceph file system as needed. We attempted an in place upgrade to 0.21 yesterday. Upon restarting the ceph processes on the MDS/monitoring node, we could no longer login to ceph due to cephx authentication failure. We did not change the configuration in /etc/ceph/ceph.conf. I was able to disable cephx, and eventually imported some keys and put the admin_keyring.bin file in /etc/ceph/keyring.bin and I can login now. Was the code changed to enforce cephx or stop looking in certain locations for keys? I believe it was enabled before, but it never complained. Also, what causes an MDS to be blacklisted? Until I fixed the authentication issues, "ceph mds stat" returned "3 blacklisted MDS". We only have 1 MDS, so I was curious what this error message meant. If I disabled cephx and restarted the MDS/monitoring services, it didn't show any blacklisted MDS. I assume blacklisting is related to an unauthenticated MDS in the cluster?? If this documentation exists, I apologize for these questions. I was interested in more detailed information on cephx (where is the auth database stored) and what the various keyring lines in the ceph.conf file control. Also, is there a baseline sample of what your "ceph auth list" output should look like and the minimum capabilities various entities need to function? I pasted a redacted version of our "auth list" output below. Perhaps we have an error in our authorization list that caused the errors we experienced after the upgrade (and before I changed some things). I can send a ceph.conf file as well if you need it. We can provide feedback on cephx code if you need it, as we were planning on keeping our cluster "cephx enabled". Thanks, Nathan Regola Grid and Cloud Computing Analyst University of Notre Dame Center for Research Computing P.O. Box 539 Room 110 Information Technology Center Notre Dame, IN 46556 Phone: 574-631-5287 10.07.30_13:07:08.584179 mon <- [auth,list] 10.07.30_13:07:08.584825 mon0 -> 'installed auth entries: mon.0 key: T caps: [mon] allow * mds.opteron03 key: U caps: [mds] allow caps: [mon] allow rwx caps: [osd] allow * osd.0 key: V caps: [mon] allow rwx caps: [osd] allow * osd.1 key: W caps: [mon] allow rwx caps: [osd] allow * client.admin key: X caps: [mds] allow caps: [mon] allow * caps: [osd] allow * ' (0) -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
