Re: [Pvfs2-developers] Distribution by hostname

2006-10-09 Thread Julian Martin Kunkel
Hi, > So, the plan is to put all of the hints in one long string and just pass > that string in through the create interface? not necessarily, I think it could be seperated also. I think the example I gave was similar to the way of passing in distribution parameters into MPI-IO some guys have imp

Re: [Pvfs2-developers] duplicate entries in directory listing

2006-10-09 Thread Murali Vilayannur
Hi, Phil, attached patch fixes the behavior that you described. Basically the problem is exactly what Sam had described. There is no point retrying a readdir of 32 entries when the actual ls command may have issued previous readdir's that are not retried. That said, I am sure this will cause othe

Re: [Pvfs2-developers] duplicate entries in directory listing

2006-10-09 Thread Sam Lang
On Oct 9, 2006, at 3:23 PM, Phil Carns wrote: Phil Carns wrote: I started thinking about some more possible ideas, but I realized after looking closer at the code that I don't actually see why duplicates would occur in the first place with the algorithm that is being used :) I apologize

Re: [Pvfs2-developers] duplicate entries in directory listing

2006-10-09 Thread Sam Lang
On Oct 9, 2006, at 3:23 PM, Phil Carns wrote: Phil Carns wrote: I started thinking about some more possible ideas, but I realized after looking closer at the code that I don't actually see why duplicates would occur in the first place with the algorithm that is being used :) I apologize

Re: [Pvfs2-developers] Distribution by hostname

2006-10-09 Thread Sam Lang
On Oct 9, 2006, at 12:05 PM, Julian Martin Kunkel wrote: Hi, So I think we're mostly trying to work out what our API should really be, whether we should extend the distro functionality vs. going totally to hints, and if we go to hints what that API should look like, right? probably the dis

Re: [Pvfs2-developers] duplicate entries in directory listing

2006-10-09 Thread Phil Carns
Phil Carns wrote: I started thinking about some more possible ideas, but I realized after looking closer at the code that I don't actually see why duplicates would occur in the first place with the algorithm that is being used :) I apologize if this has been discussed a few times already, b

Re: [Pvfs2-developers] duplicate entries in directory listing

2006-10-09 Thread Phil Carns
I started thinking about some more possible ideas, but I realized after looking closer at the code that I don't actually see why duplicates would occur in the first place with the algorithm that is being used :) I apologize if this has been discussed a few times already, but could we walk th

Re: [Pvfs2-developers] duplicate entries in directory listing

2006-10-09 Thread Phil Carns
We've talked about having pvfs2-client pull out duplicates (or the kernel module) in the cases where one of those chooses to break a readdir into multiple operations, but we haven't spent much time investigating where the replication is actually happening in order to accomplish this. Othe

Re: [Pvfs2-developers] Distribution by hostname

2006-10-09 Thread Bradley W Settlemyer
Julian Martin Kunkel wrote: Hi, So I think we're mostly trying to work out what our API should really be, whether we should extend the distro functionality vs. going totally to hints, and if we go to hints what that API should look like, right? probably the distribution needs a function which i

Re: [Pvfs2-developers] Distribution by hostname

2006-10-09 Thread Walter B. Ligon III
So, the plan is to put all of the hints in one long string and just pass that string in through the create interface? Walt Julian Martin Kunkel wrote: Hi, So I think we're mostly trying to work out what our API should really be, whether we should extend the distro functionality vs. going tot

Re: [Pvfs2-developers] Distribution by hostname

2006-10-09 Thread Julian Martin Kunkel
Hi, >So I think we're mostly trying to work out what our API should really >be, whether we should extend the distro functionality vs. going totally >to hints, and if we go to hints what that API should look like, right? probably the distribution needs a function which initializes/sets the interna

Re: [Pvfs2-developers] Distribution by hostname

2006-10-09 Thread Walter B. Ligon III
I'm down with that. For the record, my concern over strings isn't in our processing them - its the users having to muck with them - especially when passing numeric data. I just worry about lots of mallocing/string copying blech when all you really want to do is set the stripe size to the vari

Re: [Pvfs2-developers] duplicate entries in directory listing

2006-10-09 Thread Julian Martin Kunkel
Hi, > Any other ideas? The whole issue would be resolved naturally by using the filename as key instead of the number. Also the pcache would not longer be necessary... I know I'm really persistant on this one :) Julian ___ Pvfs2-developers mailing list

Re: [Pvfs2-developers] duplicate entries in directory listing

2006-10-09 Thread Sam Lang
On Oct 9, 2006, at 11:02 AM, Rob Ross wrote: Hi Phil, That's been around for a while (years). It's an artifact of ls reading pieces of the directory list at a time, and the directory list changing between directory reads. We've talked about having pvfs2-client pull out duplicates (or the

Re: [Pvfs2-developers] duplicate entries in directory listing

2006-10-09 Thread Rob Ross
Hi Phil, That's been around for a while (years). It's an artifact of ls reading pieces of the directory list at a time, and the directory list changing between directory reads. We've talked about having pvfs2-client pull out duplicates (or the kernel module) in the cases where one of those c

Re: [Pvfs2-developers] Distribution by hostname

2006-10-09 Thread Rob Ross
I'm warming to the idea of doing everything related to distributions through the hints mechanism, even though it sort of undoes Pete's attempt to simplify the discussion. The cost of string processing is minimal compared to everything else that we do, so that isn't a big deal IMO. We could sti

Re: [Pvfs2-developers] duplicate entries in directory listing

2006-10-09 Thread Sam Lang
This has got to caused by the way I did the caching of positions on the server. I think it might make sense to replace that with code that uses the component name as the position, instead of trying to debug this problem. I feel like the caching is the inherent problem that causes these

[Pvfs2-developers] duplicate entries in directory listing

2006-10-09 Thread Phil Carns
We are seeing a strange bug where if we list the contents of a directory while files are being created in it, we sometimes get duplicates and/or missing files in the output. I can reproduce it on a single machine by running these two scripts at the same time: tester.sh: -

Re: [Pvfs2-developers] Distribution by hostname

2006-10-09 Thread Walter B. Ligon III
I tend to agree with Pete and Sam ... Given the existing distro interface this is the right place to set a list of specific servers - though I tend to think it should be independent of the distro type (just as the number of servers is currently independent of which distro type you use). The li