Thanks Tim,
You are correct the /mnt/lustre is the lustre share, and /share/mnt is
an NFS share where the CTDB_RECOVERY_LOCK file is stored for the 3 ctdb
services running on 3 clients.
I think that the problem could have something to do with the
--with-privatedir
I wasn't sure what 'private dir' should point to in a cluster
environment.
My ctdb tdb's are located at /usr/local/var/ctdb/
My Samba secret.tdb is located at /etc/samba/ so should the
--with-privatedir point to /etc/samba ?
And am I correct in using the following when configuring samba
--with-lockdir=/usr/local/var/ctdb ?
Regards
Darren George
Linux Server Administrator |
 |
CONFIDENTIALITY: This e-mail
and any attachment is confidential. If you
are not the intended recipient or you receive this e-mail in error,
please notify the sender immediately and do not disclose the contents
to another person or take or print copies or use it for any other
purpose, but delete it immediately.
VIRUSES: We offer no guarantees that this e-mail or any attachment will
be received error or virus free and accept no responsibility whatever
in this regard.
SECURITY: This transmission is not guaranteed as secure. We do not
accept liability for any errors or omissions in the contents of this
e-mail which results from this transmission.
Registered in England under number 3579805. Registered address as above
|
Tim Wickberg wrote:
It's a
little hard to tell, but I'm assuming that /mnt/lustre is the lustre
mount, and /share/mnt is either kept to each machine (but the name
seems to imply that it isn't) or is shared storage that all three
machines are seeing (but that isn't the lustre pool?).
Either way, I'm guessing that either:
- Your --private-dir isn't very private (and is on shared storage at
/share/mnt/ )
- Or, your CTDB_RECOVERY_LOCK is too private.
The --with-privatedir directory needs to be local to the machine - if
all three ctdb's are trying to maintain their local state on a shared
FS things would probably not behave normally.
CTDB_RECOVERY_LOCK need to be on shared storage (preferably on the
lustre pool) to have the three ctdbd's communicate. (Is ctdbd running
on each? Any debug messages from it?)
You may also want to watch out for
https://bugzilla.lustre.org/show_bug.cgi?id=5135
Hope this helps,
- Tim
Darren George wrote:
Hi,
I'm having an issue with file locking whist using lustre+ctdb+samba
My Lustre Setup
mds/mgs
3 oss
3 Clients each with ctdb and samba services running.
All servers and clients running Centos 5 - Kernal 2.6.19-92
All servers and clients with Lustre 1.6.6
ctdb 3.0
Samba 3.2.1
CTDB Configuration:
./configure --bindir=/usr/bin --sbindir=/usr/sbin
--libdir=/usr/lib/ctdb
/etc/sysconfig/ctdb
CTDB_RECOVERY_LOCK="/share/mnt/recovery"
CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses
CTDB_MANAGES_SAMBA=yes
CTDB_LOGFILE=/var/log/log.ctdb
/etc/ctdb/public_addresses
192.168.3.200/24 eth0
/etc/ctdb/nodes
192.168.3.173
192.168.2.3
192.168.1.3
SAMBA Configuration
./configure --with-ctdb=/etc/ctdb --with-cluster-support
--enable-pie=no --bindir=/usr/bin --sbindir=/usr/sbin
--libdir=/usr/lib/samba --with-privatedir=/share/mnt/ctdb
--with-lockdir=/usr/local/var/ctdb --with-piddir=/var/run
--with-swatdir=/usr/share/swat --with-logfilebase=/var/log/samba
--with-configdir=/etc/samba
Load smb config files from /etc/samba/smb.conf
Processing section "[homes]"
Processing section "[filepool]"
Processing section "[printers]"
Loaded services file OK.
Server role: ROLE_STANDALONE
Press enter to see a dump of your service definitions
[global]
workgroup = MYGROUP
server string = Samba Server Version %v
passdb backend = tdbsam
use mmap = No
clustering = Yes
idmap backend = tdb2
fileid:mapping = fsname
ea support = Yes
cups options = raw
[filepool]
comment = The Shared Area
path = /mnt/lustre
read
create mask = 0777
directory mask = 0777
All lustre clients have been mounted with the '-o flock' option.
Everything from a linux prospective is working fine, my linux clients
can read/write to files on the file share and file locking is active.
>From a window share prospective clients can read/write to files on
the file share but file locking is not active.
The only errors I can find from samba logs are:
[2009/01/11 02:50:25, 3]
locking/locking.c:fetch_share_mode_unlocked(857)
fill_share_mode_lock failed
[2009/01/11 02:50:25, 10] smbd/statcache.c:stat_cache_lookup(235)
stat_cache_lookup: lookup failed for name [SG]
[2009/01/11 02:50:33, 5] lib/util_sock.c:read_socket_with_timeout(928)
read_socket_with_timeout: blocking read. EOF from client.
[2009/01/11 02:50:33, 3] smbd/process.c:smbd_process(2027)
receive_message_or_smb failed: NT_STATUS_END_OF_FILE, exiting
Has anyone dealt with a similar set-up and can assist with the file
locking from a ctdb+samba environment ?
|
_______________________________________________
Lustre-discuss mailing list
[email protected]
http://lists.lustre.org/mailman/listinfo/lustre-discuss
