[Samba] smbd reads entire directory when creating files?
Hi, http://lists.samba.org/archive/samba/2004-December/096861.html Was this problem solved? If yes, what way? Your assistance will be greatly appreciated! Thanks in advance. -- best regards, Oleg Laktionov -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/listinfo/samba
Re: [Samba] smbd reads entire directory when creating files?
Jeremy Allison wrote: On Wed, Dec 08, 2004 at 01:37:23PM -0600, Matt Mitchell wrote: Obviously such a special case would make the code ugly...but I might try patching it just for my own testing to see if it makes any difference. Any pointers you can offer? That's exactly the case I was intending to add :-). I'm have to work on the malloc issue at the moment - you seem to have quickly identified the neccessary optimization without my help :-) - well done ! Don't give me too much credit just yet :-) So it appears that unix_convert behaves reasonably well if conn-case_sensitive is true (and mangling is not in effect, which I would guess is true in probably 95% of cases today). There is already an escape in there which is triggered if the user is not so dumb as to assume that he is using the share he thinks he is (with case sensitive = yes) as opposed to one without. filename.c:186 in my sources. That takes care of one readdir. The other is happening as a result of either xcopy or Windows checking to see if the file exists before copying it. i.e. it's doing a call_trans2findfirst, which calls OpenDir eventually, reading in the whole directory before deciding that the file isn't already there. This one is more subtle, and I don't know enough about the SMB semantics to really know what a valid approach would be. The same logic for optimization would seem to apply whenever conn-case_sensitive is in effect and we are not asked for a wildcard string, but I have no idea about implications for other areas of the protocol that might depend on that dirptr actually being populated. As I have time today I will keep digging. Any insight appreciated. Should this be moved to samba-technical? -m -- To unsubscribe from this list go to the following URL and read the instructions: http://lists.samba.org/mailman/listinfo/samba
Re: [Samba] smbd reads entire directory when creating files?
Jeremy Allison wrote: A strace on the smbd process receiving the files from the windows box (it is mapped as a share on the 2k3 server) reveals that smbd is looking up the entire directory (with getdents64) every time it writes a file. Several times, in fact. So as the number of files grows, it churns more and more. I'm sure most of it is in cache but the data still has to be moved around in memory. I know this is not necessary for writing/copying files with unix semantics, but I wonder if the case-fiddling or any other Windows imitative behavior is making this getdents64 orgy necessary. Is there any way to disable it? I was also looking at the case sensitivity options, but alas they don't seem to be able to prevent the readdir bonanza. I'm guessing the unix_convert routine is responsible for at least one set of traversals. In this case I don't care at all about converting these paths and there are no wildcards. It seems that, in that particular case, the contract of the routine could be satisfied by checking to see if we are in case sensitive mode and if so we don't bother doing the scan_directory (filename.c:284), since the SMB_VFS_STAT will tell us if the file really exists or not. Granted, it's an optimization. I realize that in the presence of wildcards (of which case-insensitivity is a variety) you have to do that scan_directory call. Of course, ideally, you would cache that resulting directory list as long as you possibly can. Name mangling also complicates this, but it's another feature I'm not using at all in this application. Obviously such a special case would make the code ugly...but I might try patching it just for my own testing to see if it makes any difference. Any pointers you can offer? -m -- To unsubscribe from this list go to the following URL and read the instructions: http://lists.samba.org/mailman/listinfo/samba
Re: [Samba] smbd reads entire directory when creating files?
On Wed, Dec 08, 2004 at 01:37:23PM -0600, Matt Mitchell wrote: Obviously such a special case would make the code ugly...but I might try patching it just for my own testing to see if it makes any difference. Any pointers you can offer? That's exactly the case I was intending to add :-). I'm have to work on the malloc issue at the moment - you seem to have quickly identified the neccessary optimization without my help :-) - well done ! Jeremy. -- To unsubscribe from this list go to the following URL and read the instructions: http://lists.samba.org/mailman/listinfo/samba
[Samba] smbd reads entire directory when creating files?
Apologies if this is a FAQ.
We are saddled with some directories full of production image files that
number in the hundreds of thousands to the millions. (Yes, I know this
is bad.) They reside on a couple of Win 2003 boxes that are flaky and
unreliable. I was hoping to copy them over to a Linux box running samba
but it is taking forever.
A strace on the smbd process receiving the files from the windows box
(it is mapped as a share on the 2k3 server) reveals that smbd is looking
up the entire directory (with getdents64) every time it writes a file.
Several times, in fact. So as the number of files grows, it churns more
and more. I'm sure most of it is in cache but the data still has to be
moved around in memory.
I know this is not necessary for writing/copying files with unix
semantics, but I wonder if the case-fiddling or any other Windows
imitative behavior is making this getdents64 orgy necessary. Is there
any way to disable it?
I was going to post a bit of the strace output but it is extremely long
and (to my eyes, anyway) not very interesting. I can supply it to
anyone who is interested in seeing it.
Any suggestions appreciated. I also note that while using smbclient
instead of mapping the drive from the Windows box might help with this
copy, it will not help if we were to try to use this setup to replace
the 2k3 servers (the eventual goal). So any pointers in that direction,
while perhaps interesting for comparison, are not especially relevant.
greenville:~# smbd -V
Version 3.0.7-Debian
smb.conf is attached. Not much interesting in there, it is pretty much
stock. vfs_ownerwrite is a small VFS module I hacked up to do an
extra chown on a file that is written to the share (used in a common
PDC/Unix environment on some shared directories.) I am using the
deadwood_ftp share in this case, so the module is not involved.
-m
# Samba configuration file for media servers.
# -
#
# $Id: smb.conf,v 1.2.4.6 2002/03/13 18:56:16 peloy Exp $
#
# This is the main Samba configuration file. You should read the
# smb.conf(5) manual page in order to understand the options listed
# here. Samba has a huge number of configurable options most of which
# are not shown in this example
#
# Any line which starts with a ; (semi-colon) or a # (hash)
# is a comment and is ignored. In this example we will use a #
# for commentary and a ; for parts of the config file that you
# may wish to enable
#
# NOTE: Whenever you modify this file you should run the command
# testparm to check that you have not many any basic syntactic
# errors.
#
#=== Global Settings ===
[global]
# can this be turned back on?
use sendfile = no
# Do something sensible when Samba crashes: mail the admin a backtrace
panic action = /usr/share/samba/panic-action %d
# Change this for the workgroup/NT-domain name your Samba server will part of
workgroup = VP
# server string is the equivalent of the NT Description field
server string = %h server (Samba %v)
# If you want to automatically load your printer list rather
# than setting them up individually then you'll need this
; load printers = yes
# You may wish to override the location of the printcap file
; printcap name = /etc/printcap
# 'printing = cups' works nicely
; printing = bsd
; guest account = nobody
invalid users = root
# This tells Samba to use a separate log file for each machine
# that connects
log file = /var/log/samba/log.%m
# Put a capping on the size of the log files (in Kb).
max log size = 1000
# If you want Samba to log though syslog only then set the following
# parameter to 'yes'. Please note that logging through syslog in
# Samba is still experimental.
; syslog only = no
# We want Samba to log a minimum amount of information to syslog. Everything
# should go to /var/log/samba/log.{smb,nmb} instead. If you want to log
# through syslog you should set the following parameter to something higher.
syslog = 0
# security = user is always a good idea. This will require a Unix account
# in this server for every user accessing the server. See
# security_level.txt for details.
security = domain
# You may wish to use password encryption. Please read ENCRYPTION.txt,
# Win95.txt and WinNT.txt in the Samba documentation. Do not enable this
# option unless you have read those documents
encrypt passwords = true
passdb backend = tdbsam guest
# Using the following line enables you to customise your configuration
# on a per machine basis. The %m gets replaced with the netbios name
# of the machine that is connecting
; include = /home/samba/etc/smb.conf.%m
# Most people will find that this option gives better performance.
# See speed.txt and the manual pages for details
# You may want to add the following on a Linux system:
# SO_RCVBUF=8192 SO_SNDBUF=8192
socket options = TCP_NODELAY
# --- Browser Control Options ---
# Please _read_ BROWSING.txt and set the
Re: [Samba] smbd reads entire directory when creating files?
On Tue, Dec 07, 2004 at 02:08:56PM -0600, Matt Mitchell wrote: Apologies if this is a FAQ. Kind of. We are saddled with some directories full of production image files that number in the hundreds of thousands to the millions. (Yes, I know this is bad.) They reside on a couple of Win 2003 boxes that are flaky and unreliable. I was hoping to copy them over to a Linux box running samba but it is taking forever. This is pretty much the worse case scenario for Samba I'm afraid. A strace on the smbd process receiving the files from the windows box (it is mapped as a share on the 2k3 server) reveals that smbd is looking up the entire directory (with getdents64) every time it writes a file. Several times, in fact. So as the number of files grows, it churns more and more. I'm sure most of it is in cache but the data still has to be moved around in memory. I know this is not necessary for writing/copying files with unix semantics, but I wonder if the case-fiddling or any other Windows imitative behavior is making this getdents64 orgy necessary. Is there any way to disable it? Not currently. This is something I'm working on - would you be willing to test some prototype code once I'm done ? Jeremy. -- To unsubscribe from this list go to the following URL and read the instructions: http://lists.samba.org/mailman/listinfo/samba
Re: [Samba] smbd reads entire directory when creating files?
Jeremy Allison wrote: On Tue, Dec 07, 2004 at 02:08:56PM -0600, Matt Mitchell wrote: A strace on the smbd process receiving the files from the windows box (it is mapped as a share on the 2k3 server) reveals that smbd is looking up the entire directory (with getdents64) every time it writes a file. Several times, in fact. So as the number of files grows, it churns more and more. I'm sure most of it is in cache but the data still has to be moved around in memory. I know this is not necessary for writing/copying files with unix semantics, but I wonder if the case-fiddling or any other Windows imitative behavior is making this getdents64 orgy necessary. Is there any way to disable it? Not currently. This is something I'm working on - would you be willing to test some prototype code once I'm done ? I'd love to. This file count also exposes some brokenness in the 'tar' functionality of smbclient, perhaps in smbclient itself (my next workaround attempt). I'll try to gather more info on that for whoever might be interested, or patch it myself if possible. -m -- To unsubscribe from this list go to the following URL and read the instructions: http://lists.samba.org/mailman/listinfo/samba
