Re: Utilizing OSDL's STP for Reiser4 testing/benchmarks?

2006-07-21 Thread Hans Reiser
Nate Diller wrote:

> On 7/21/06, Mike Benoit <[EMAIL PROTECTED]> wrote:
>
>> Has the Reiser4 team looked at utilizing the OSDL Scalable Test Platform
>> (STP) service to benchmark and test Reiser4 patches? They seem to offer
>> a wide variety of hardware to test on, and already have some file system
>> benchmarks available to choose from.
>>
>> I noticed the Namesys benchmark page hasn't been updated in quite some
>> time, understandably so, as benchmarking can be incredibly time
>> consuming.  Perhaps using the STP would make this much easier to do more
>> frequently?
>>
>> It would be really nice to see how the batch_write/read(?) patches to
>> the kernel affect things, and how Reiser4 performance progresses over
>> time. I know Hans has mentioned CPU usage dropping in recent patches,
>> but actual numbers from a benchmark are usually more compelling to see.
>
>
> the dev team has seen a number of benchmarks which Hans' statements
> are based on... maybe we should post some of them publicly?
>
> NATE
>
>
Sure.  I asked for that, actually.


Re: reiser4 status (correction)

2006-07-21 Thread Hans Reiser
David Masover wrote:

>
>
> And it's not just databases.  Consider BitTorrent.  The usual
> BitTorrent way of doing things is to create a sparse file, then fill
> it in randomly as you receive data.  Only if you decide to allocate
> the whole file right away, instead of making it sparse, you gain
> nothing on Reiser4, since writes will be just as fragmented as if it
> was sparse.

If you don't flush it before you fill it, then in reiser4 it does not
matter if it starts out sparse.

>
> Personally, I'd rather leave it as sparse, but repack everything later.

We have 3 levels of optimization: 1) at each modification, 2) at each
flush, and 3) at each repack.  Each of these operates on a different
time scale, and all 3 are worthy of doing as right as we can.

Now, the issue of where should airholes be?  Why, that is the grand
experiment that will start to happen in a few months.  Nobody knows yet
what defaults we should have, and whatever we choose, there will be some
users who gain from explicit control of it.

>   it must not be as trivial as I think it is.

The problem is that there was a list of must dos, and this was just one
of them.  If reiser4 goes in, then fsync is the only thing in front of
the repacker.  The list has reduced in size a bunch.

>
>> A much better approach in my opinion would be to have Reiser4 perform
>> well in the majority of cases without the repacker, and sell the
>> repacker to people who need that extra bit of performance. If I'm not
>> mistaken this is actually Hans intent.
>
>
> Hans?

Yes, that's the idea.  Only sysadmins of large corps are likely to buy. 
We throw in service and support as well for those who purchase it.

If I was making money, I would not do this, but I am not.  I am not
really willing to work a day job for the rest of my life supporting guys
in Russia, it is only ok to do as a temporary measure.  I am getting
tired

>
>> If Reiser4 does turn out to
>> perform much worse over time, I would expect Hans would consider it a
>> bug or design flaw and try to correct the problem however possible. 
>
>
I would want Reiser4 without a repacker to outperform all other
filesystems.  The problem with this fragmentation over time issue is
that it is hard to tweak allocation, measure the effect, tweak again,
etc.  Not sure how to address it without a lot of work.  Maybe we need
to create some sort of condensed portage benchmark

Hans

Hans


Re: Utilizing OSDL's STP for Reiser4 testing/benchmarks?

2006-07-21 Thread Nate Diller

On 7/21/06, Mike Benoit <[EMAIL PROTECTED]> wrote:

Has the Reiser4 team looked at utilizing the OSDL Scalable Test Platform
(STP) service to benchmark and test Reiser4 patches? They seem to offer
a wide variety of hardware to test on, and already have some file system
benchmarks available to choose from.

I noticed the Namesys benchmark page hasn't been updated in quite some
time, understandably so, as benchmarking can be incredibly time
consuming.  Perhaps using the STP would make this much easier to do more
frequently?

It would be really nice to see how the batch_write/read(?) patches to
the kernel affect things, and how Reiser4 performance progresses over
time. I know Hans has mentioned CPU usage dropping in recent patches,
but actual numbers from a benchmark are usually more compelling to see.


the dev team has seen a number of benchmarks which Hans' statements
are based on... maybe we should post some of them publicly?

NATE


Re: reiser4 status (correction)

2006-07-21 Thread David Masover

Mike Benoit wrote:

Your detailed explanation is appreciated David and while I'm far from a
file system expert, I believe you've overstated the negative effects
somewhat.

It sounds to me like you've gotten Reiser4's allocation process in
regards to wandering logs correct, from what I've read anyways, but I
think you've overstated its fragmentation disadvantage when compared
against other file systems.

I think the thing we need to keep in mind here is that fragmentation
isn't always a net loss. Depending on the workload, fragmentation (or at
least not tightly packing data) could actually be a gain. In cases where


defragmented != tightly packed.


you have files (like log files or database files) that constantly grow
over a long period of time, packing them tightly at regularly scheduled
intervals (or at all?) could cause more harm then good. 


This is true...


Consider this scenario of two MySQL tables having rows inserted to each
one simultaneously, and lets also assume that the two tables were
tightly packed before we started the insert process.

1 = Data for Table1
2 = Data for Table2 


Tightly packed:



Simultaneous inserts start:

1122112211221122



Allocate on flush alone would probably help this scenario immensely. 


Yes, it would.  You'd end up with



assuming they both fit into RAM.  And of course they could later be 
repacked.


By the way, this is the NTFS approach to avoiding fragmentation -- try 
to avoid fragmenting anything below a certain block size.  I, for one, 
would be perfectly happy if my large files were split up every 50 or 100 
megs or so.


The problem is when you get tons of tiny files and metadata stored so 
horribly inefficiently that things like Native Command Queuing is 
actually a huge performance boost.



The other thing you need to keep in mind is that database files are like
their own little mini-file system. They have their own fragmentation
issues to deal with (especially PostgreSQL).


I'd rather not add to that.  This is one reason to hate virtualization, 
by the way -- it's bad enough to have a fragmented NTFS on your Windows 
installation, but worse if the disk itself is a fragmented sparse file 
on Linux.



So in cases like you
described where you are overwriting data in the middle of a file,
Reiser4 may be poor at doing this specific operation compared to other
file systems, but just because you overwrite a row that appears to be in
the middle of a table doesn't mean that the data itself is actually in
the middle of the table. If your original row is 1K, and you try to
overwrite it with 4K of data, it most likely will be put at the end of
the file anyways, and the original 1K of data will be marked for
overwriting later on. Isn't this what myisampack is for?


If what you say is true, isn't myisampack also an issue here?  Surely it 
doesn't write out an entirely separate copy of the file?


Anyway, the most common usage I can see for mysql would be overwriting a 
1K row with another 1K row, or dropping a row, or adding a wholly new 
row.  I may be a bit naive here...


But then, isn't there also some metadata somewhere which says things 
like how many rows you have in a given table?


And it's not just databases.  Consider BitTorrent.  The usual BitTorrent 
way of doing things is to create a sparse file, then fill it in randomly 
as you receive data.  Only if you decide to allocate the whole file 
right away, instead of making it sparse, you gain nothing on Reiser4, 
since writes will be just as fragmented as if it was sparse.


Personally, I'd rather leave it as sparse, but repack everything later.


So while I think what you described is ultimately correct, I believe
extreme negative effects from it to be a corner case, and probably not
representative of the norm. I also believe that other Reiser4
improvements would outweigh this draw back to wandering logs, again in
average workloads. 


Depends on your definition of average.  I'm also speaking from 
experience.  On Gentoo, /usr/portage started out being insanely fast on 
Reiser4, because it barely had to seek at all -- despite being about 
145,000 small files.  I think it was maybe half that when I first put it 
on r4, but it's more than twice as slow now, and you can hear it thrashing.


Now, the wandering logs did make the rsync process pretty fast -- the 
entire thing gets rsync'd against one of the Gentoo mirrors.  For anyone 
using Debian, this is the equivalent of "apt-get update".


Only now, this rsync process is not only entirely disk-bound, it's 
something like 10x as slow.  I have a gig of RAM, so at least it's fast 
once it's cached, but it's obviously horrendously fragmented.  I am not 
sure if it's individual files or directories, but it could REALLY use a 
repack.


From what I remember of v3, it was never quite this bad, but it never 
started o

Re: storing images thumbnails as pseudo files?

2006-07-21 Thread Hans Reiser
Pysiak Satriani wrote:

>Hello,
>
>suppose pseudo files, file-as-directory are on my r4 partition and are usuable.
>
>Does namesys' vision allow things like storing image thumbnails "inside the 
>file
>itself" ?
>
>Example using jpgtn (jpgtn creates thumbnails of jpg files)
>
>// create 150px thumbnail
>$ jpgtn -s 150 -H test.jpg
>
>// move the thumbnail "into" the file
>$ mv tn_test test.jpg/thumbs/150px
>
>// the same but a 100px thumbnail
>$ jpgtn -s 100 -H test.jpg
>$ mv tn_test test.jpg/thumbs/100px
>
>Then I could write HTML code like this:
>
>
>
>This idea came to me because of a project that has to create many thumbnails
>for many files. That requires to make a naming scheme, parse filenames and
>don't make one mistake. If I could do things this way it would be much
>easier and elegant to do.
>
>But that's just an idea from someone not really knowledgible enough in
>the area.
>
>Best regards,
>Maciej Soltysiak
>
>
>
>
>  
>
What you describe is the vision we are trying for.


Re: somewhat OT query on journalling

2006-07-21 Thread Hans Reiser
David Masover wrote:

> Andreas Schäfer wrote:
>
>>> Don't get too excited -- the transactions probably aren't done yet.
>>> Without those, no filesystem that claims to journal data is really
>>> any better than a filesystem which only journals metadata.  Even
>>> once they are implemented (or even if they are already),
>>> applications have to support them directly.
>>
>>
>> Actually, I think transactions in a filesystem context are a bit
>> different from the transactions you know form databases. Generally
>
>
> Yes, generally speaking, you're entirely right.  But in the case of
> Reiser4, at least for a single file, you can perform a number of
> writes and declare them a single transaction.

If we finish that code you can.;-)  One of the problems that we need to
deal with is that we are shipping a product pared of all functionality
not essential so that we can get it out the door, and the website still
describes the whole vision.  We will do the whole vision, but first we
need to get some income flowing.



Re: reiser4 status (correction)

2006-07-21 Thread Hans Reiser
Mike Benoit wrote:

>
>On top of that, I don't see how a repacker would help these work loads
>much as the files usually have a high churn rate. 
>
I think Reiserfs is used on a lot more than squid servers.  For them,
80% of files don't move for long periods of time is the usual industry
statistic


Re: storing images thumbnails as pseudo files?

2006-07-21 Thread Hans Reiser
David Masover wrote:

> Disclaimer:  I don't speak for Namesys, and I don't work here.  While
> I'm pretty confident I understand their vision, the final word on
> anything Reiser is always from Hans Reiser.

David described my views pretty well, and saved me much typing.:)


Re: reiser4 status (correction)

2006-07-21 Thread Mike Benoit
Your detailed explanation is appreciated David and while I'm far from a
file system expert, I believe you've overstated the negative effects
somewhat.

It sounds to me like you've gotten Reiser4's allocation process in
regards to wandering logs correct, from what I've read anyways, but I
think you've overstated its fragmentation disadvantage when compared
against other file systems.

I think the thing we need to keep in mind here is that fragmentation
isn't always a net loss. Depending on the workload, fragmentation (or at
least not tightly packing data) could actually be a gain. In cases where
you have files (like log files or database files) that constantly grow
over a long period of time, packing them tightly at regularly scheduled
intervals (or at all?) could cause more harm then good. 

Consider this scenario of two MySQL tables having rows inserted to each
one simultaneously, and lets also assume that the two tables were
tightly packed before we started the insert process.

1 = Data for Table1
2 = Data for Table2 

Tightly packed:



Simultaneous inserts start:

1122112211221122

I believe this is actually what is happening to me with ReiserV3 on my
MythTV box. I have two recordings running at the same time, each writing
data at about 500kb/s and once the drive has less then 10% free the
whole machine grinds to a screeching halt while it attempts to find free
space. The entire 280GB drive is full of files fragmented like this.
 
Allocate on flush alone would probably help this scenario immensely. 

The other thing you need to keep in mind is that database files are like
their own little mini-file system. They have their own fragmentation
issues to deal with (especially PostgreSQL). So in cases like you
described where you are overwriting data in the middle of a file,
Reiser4 may be poor at doing this specific operation compared to other
file systems, but just because you overwrite a row that appears to be in
the middle of a table doesn't mean that the data itself is actually in
the middle of the table. If your original row is 1K, and you try to
overwrite it with 4K of data, it most likely will be put at the end of
the file anyways, and the original 1K of data will be marked for
overwriting later on. Isn't this what myisampack is for?

So while I think what you described is ultimately correct, I believe
extreme negative effects from it to be a corner case, and probably not
representative of the norm. I also believe that other Reiser4
improvements would outweigh this draw back to wandering logs, again in
average workloads. 

So the original point that I was trying to make comes back to the fact
that I don't believe Reiser4 _needs_ a repacker to maintain decent
performance. The fact that it will have a repacker just makes it that
much better for people who might need it. If Hans didn't think he could
make money off it, it probably wouldn't be so high on his priority list?
We can't fault him for that though.

Like you mentioned, if Reiser4 performance gets so poor without the
repacker, and Hans decides to charge for it, I think that will turn away
a lot potential users as they could feel that this is a type of
extortion. Get them hooked on something that only performs well for a
certain amount of time, then charge them money to keep it up. I also
think the community would write their own repacker pretty quick in
response. 

A much better approach in my opinion would be to have Reiser4 perform
well in the majority of cases without the repacker, and sell the
repacker to people who need that extra bit of performance. If I'm not
mistaken this is actually Hans intent. If Reiser4 does turn out to
perform much worse over time, I would expect Hans would consider it a
bug or design flaw and try to correct the problem however possible. 

But I guess only time will tell if this is true or not. ;)

On Fri, 2006-07-21 at 17:40 -0500, David Masover wrote:
> Maybe not XFS, but in any case, Reiser4 fragments more because of how 
> its journaling works.  It's the wandering logs.
> 
> Basically, when most Linux filesystems allocate space, they do try to 
> allocate it contiguously, and it generally stays in the same place. 
> With ext3, if you write to the middle of a file, or overwrite the entire 
> file, you're generally going to see your writes be written once to the 
> journal, and then again to the same place the file originally was.
> 
> Similarly, if you delete and then create a bunch of small files, you're 
> generally going to see the new files created in the same place the old 
> files were.
> 
> With Reiser4, wandering logs means that rather than write to the 
> journal, if you write to the middle of the file, it writes that chunk to 
> somewhere else on the disk, and somehow gets it down to one atomic 
> operation where it simply changes the file to point to the new location 
> on disk.  Which means if you have a filesystem

Re: reiser4 status (correction)

2006-07-21 Thread Andreas Schäfer
On 17:45 Fri 21 Jul , David Masover wrote:
> Question, then:  Can the ext2 defrag work on a raw ext3 partition, without 
> having to convert it first?

Dunno, but I don't think so


pgp3cS9WVSQCi.pgp
Description: PGP signature


Re: reiser4 status (correction)

2006-07-21 Thread David Masover

Andreas Schäfer wrote:

On 14:37 Fri 21 Jul , Mike Benoit wrote:

No Linux file system that I'm aware of has a defragmentor, but they DO
become fragmented, just not near as bad as FAT32 used to when MS created
their defragmentor.


Forgotten ext2? ;-) Funny thing: If your ext3 got too fragmented:
convert it back to ext2, defrag and reconvert it again to ext3. All of
this can be done "in place", i.e. without moving the data to other
partitions etc.


Well, I know an ext3 partition can be mounted, unchanged, as ext2.  Of 
course, you had to cleanly unmount as ext3 first, and make sure you 
cleanly mount/unmount as ext2 before you try to mount as ext3 again...


Question, then:  Can the ext2 defrag work on a raw ext3 partition, 
without having to convert it first?


Re: fsck.reiserfs --rebuild-tree out of disk space aborted

2006-07-21 Thread David Masover

Joel Heenan wrote:


The filesystem contains some sensitive data so I can't really do this
I'm afraid. I'm happy to run any diagnostic tools you have or provide
any more information.


I wonder if anything like this exists already...

Maybe it would work if you could provide moderated ssh access.  Still 
faster than instant messaging or IRC, and still more secure, but you get 
to choose which commands to allow, and what output to allow (or censor).


Would a tool like this be useful?  Does one already exist?

Basically, it's a man-in-the-middle attack by the server admin, maybe 
some old black hat tools would help?


Re: reiser4 status (correction)

2006-07-21 Thread David Masover

Mike Benoit wrote:

On Fri, 2006-07-21 at 16:06 -0500, David Masover wrote:

Mike Benoit wrote:


Tuning fsync will fix the last wart on Reiser4 as far as benchmarks are
concerned won't it? Right now Reiser4 looks excellent on the benchmarks
that don't use fsync often (mongo?), but last I recall the fsync
performance was so poor it overshadows the rest of the performance. It
would also probably be more useful to a much wider audience, especially
if Namesys decides to charge for the repacker.
If Namesys does decide to charge for the repacker, I'll have to consider 
whether it's worth it to pay for it or to use XFS instead.  Reiser4 
tends to become much more fragmented than most other Linux FSes -- 
purely subjective, but probably true.




I would like to see some actual data on this. I haven't used Reiser4 for
over a year, and when I did it was only to benchmark it. But Reiser4
allocates on flush, so in theory this should decrease fragmentation, not
increase it. Due to this I question what you are _really_ seeing, or if
perhaps it is a bug in the allocator? Why would XFS or any other
multi-purpose file system resist fragmentation noticeably more then
Reiser4 does.


Maybe not XFS, but in any case, Reiser4 fragments more because of how 
its journaling works.  It's the wandering logs.


Basically, when most Linux filesystems allocate space, they do try to 
allocate it contiguously, and it generally stays in the same place. 
With ext3, if you write to the middle of a file, or overwrite the entire 
file, you're generally going to see your writes be written once to the 
journal, and then again to the same place the file originally was.


Similarly, if you delete and then create a bunch of small files, you're 
generally going to see the new files created in the same place the old 
files were.


With Reiser4, wandering logs means that rather than write to the 
journal, if you write to the middle of the file, it writes that chunk to 
somewhere else on the disk, and somehow gets it down to one atomic 
operation where it simply changes the file to point to the new location 
on disk.  Which means if you have a filesystem that is physically laid 
out on disk like this (for simplicity, assume it only has a single file):


# is data
* is also data
- is free space

##*--

When you try to write in the middle (the '*' chars) -- let's say we're 
changing them to '%' chars, this happens:


##*%-

Once that's done, the file is updated so that the middle of it points to 
the fragment in the new location, and the old location is freed:


##-%-



Keep in mind, because of lazy writes, it's much more likely for the 
whole change to happen at once.  Here's another example:


#

Let's say we just want to overwrite the file with another one of the 
same length:


#%---

then, commit the transaction:

-%---

You see the problem?  You've now split the free space in half. 
Realistically, of course, it wouldn't be by halves, but you're basically 
inserting random air holes all over the place, and your FS is becoming 
more like foam, taking up more of the free space, until you can no 
longer use the free space  In the above example, if we then have to 
come write some huge file, it looks like this:


*%***

Split right in half.  Now imagine this effect multiplied by hundreds or 
thousands of files, over time...


This is why Reiser4 needs a repacker.  While it's fine for larger files 
-- I believe after a certain point, it will write twice, so looking at 
our first example:



##*--

Write to a new, temporary place:

##*%-

Write back to the original place:

##%%-

Complete the transaction and free the temporary space:

##%--


This technique is what other journaling filesystems use, and it also 
means that writing is literally twice as slow as on a non-journaling 
filesystem, or on one with wandering logs like Reiser4.  But, it's a 
practical necessity when you're dealing with some 300 gig MySQL database 
of which only small 10k chunks are changing.  Taking twice as long on a 
10k chunk won't kill anyone, but fragmenting your 300 gig database (on a 
320 gig partition) will kill your performance, and will be very 
difficult to defragment.


But on smaller files, it would be very beneficial if we could allow the 
FS to slowly fragment (to foam-ify, if you will) and defrag once a week. 
 The amount of speed gained in each write -- and read, if it's not 
getting too awful during that week -- definitely makes up for having to 
spend an hour or so defragmenting, especially if the FS can be online at 
the time.


And you can probably figure out an optimal time to wait before 
defragmenting, since your biggest fragmentation problems happen when the 
chunk of contiguous space at the end of the di

Re: fsck.reiserfs --rebuild-tree out of disk space aborted

2006-07-21 Thread Joel Heenan

On 7/22/06, Vladimir V. Saveliev <[EMAIL PROTECTED]> wrote:

hello

On Fri, 2006-07-21 at 10:09 +1000, Joel Heenan wrote:
> Hi,
>
> I can home about two weeks ago and found my media box locked up. I was
> able to discover that it had filled up its /dev/md2 partition (mounted
> on /home) which surprised me because it is 550 gigs. Perhaps mythtv
> went nuts and used it all up. It was only a temporary thing I was
> going to move mythtv to another partition anyway and boy I wish I had
> now :-P.
>
> Anyway I rebooted and the fsck said I had to run rebuild-tree. So I
> ran that and a few days later it said "out of disk space, aborted". I
> can't mount the partition it says "No folder found I believe". I tried
> it a few times with both the reiserfsprogs from etch:
>
> ii  reiserfsprogs3.6.19-2
> User-level tools for ReiserFS filesystems
>
> and the latest ones downloaded from the website (3.6.19). I am
> currently trying it with the -S option.
>
there is no need to use -S in your case.

> I'm running a custom 2.6.12 kernel which is basically the same as the
> default debian one except it includes some drivers for my dvico fusion
> tv tuner.
>
> I read that the best way to fix this problem is to dd to a bigger
> partition but there is really no easy way for me to do that - it will
> probably involve me purchasing 2x300gig partitions, raiding them, then
> performing the restore.
>
> Please let me know if there is a way I can fix this without going to a
> bigger partition.
>
> Here is output of the fsck:
>
> """
> fsck.reiserfs --rebuild-tree /dev/md2
> reiserfsck 3.6.19 (2003 www.namesys.com)
>
> *
> ** Do not  run  the  program  with  --rebuild-tree  unless **
> ** something is broken and MAKE A BACKUP  before using it. **
> ** If you have bad sectors on a drive  it is usually a bad **
> ** idea to continue using it. Then you probably should get **
> ** a working hard drive, copy the file system from the bad **
> ** drive  to the good one -- dd_rescue is  a good tool for **
> ** that -- and only then run this program. **
> ** If you are using the latest reiserfsprogs and  it fails **
> ** please  email bug reports to reiserfs-list@namesys.com, **
> ** providing  as  much  information  as  possible --  your **
> ** hardware,  kernel,  patches,  settings,  all reiserfsck **
> ** messages  (including version),  the reiserfsck logfile, **
> ** check  the  syslog file  for  any  related information. **
> ** If you would like advice on using this program, support **
> ** is available  for $25 at  www.namesys.com/support.html. **
> *
>
> Will rebuild the filesystem (/dev/md2) tree
> Will put log info to 'stdout'
>
> Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes
> Replaying journal..
> Reiserfs journal '/dev/md2' in blocks [18..8211]: 0 transactions replayed
> ###
> reiserfsck --rebuild-tree started at Mon Jul 17 21:41:44 2006
> ###
>
> Pass 0:
> ### Pass 0 ###
> Loading on-disk bitmap .. ok, 116676032 blocks marked used
> Skipping 11771 blocks (super block, journal, bitmaps) 116664261 blocks
> will be read
> 0%20%block 85875092: The number of items (256) is incorrect,
> should be (1) - corrected
> block 85875092: The free space (1) is incorrect, should be (208) - corrected
> pass0: vpf-10110: block 85875092, item (0): Unknown item type found
> [33488896 65537 0x100 ??? (15)] - deleted
>left 0, 15065 /seccsec
> 108413 directory entries were hashed with "r5" hash.
> "r5" hash is selected
> Flushing..finished
> Read blocks (but not data blocks) 116664261
> Leaves among those 87768646
> - leaves all contents of which could not be
> saved and deleted 1
> Objectids found 108417
>
> Pass 1 (will try to insert 87768645 leaves):
> ### Pass 1 ###
> Looking for allocable blocks .. finished
> 0%20%40%..Not enough allocable blocks, checking bitmap...there
> are 1 allocable blocks, btw
>

I am working on a fix for this problem. But the above output looks
suspicious.

What kind of files are there, would you describe?


It is the /home folder on my media box. So it has around 10gigs of
personal files (normal everyday file usage), and the rest is media.
There is 40 gigs of mp3s. 100 gigs of avi files of different sizes but
mostly large at least 100 megs or more.

It also has 200 gigs of mythtv NUV files.

My suspicion is that mythtv went nuts and began recording something
and never closed the file and used up all the space. But really I
don't know because it has been unusable since I found the problem.


Are they long? Did you mount reiserfs with any options?


Some of the files are long, there would be some files that are 4 gigs
in size. Sometimes mythtv makes very large files

Re: reiser4 status (correction)

2006-07-21 Thread Andreas Schäfer
On 14:37 Fri 21 Jul , Mike Benoit wrote:
> No Linux file system that I'm aware of has a defragmentor, but they DO
> become fragmented, just not near as bad as FAT32 used to when MS created
> their defragmentor.

Forgotten ext2? ;-) Funny thing: If your ext3 got too fragmented:
convert it back to ext2, defrag and reconvert it again to ext3. All of
this can be done "in place", i.e. without moving the data to other
partitions etc.

-Andreas


pgpiwx2SW0fdR.pgp
Description: PGP signature


Re: somewhat OT query on journalling

2006-07-21 Thread David Masover

Andreas Schäfer wrote:
Don't get too excited -- the transactions probably aren't done yet. Without those, no filesystem that claims to journal data is really any better than a filesystem which only journals metadata.  Even once 
they are implemented (or even if they are already), applications have to support them directly.


Actually, I think transactions in a filesystem context are a bit
different from the transactions you know form databases. Generally


Yes, generally speaking, you're entirely right.  But in the case of 
Reiser4, at least for a single file, you can perform a number of writes 
and declare them a single transaction.


I believe there are some other oddities, such as:  The FS doesn't always 
do the rollback, sometimes it delegates to the app for that, and it may 
be possible to perform transactions on a number of files.  I don't 
believe transactions imply locking, only serialization -- that is, the 
last transaction that goes through is the one that counts, and 
overwrites any transactions that completed before, even if they were 
started after.  So:


Alice opens foo
Alice starts writing
Bob opens foo
Bob starts writing
Alice writes some more
Bob writes more
Bob closes foo
Alice closes foo

Even though Bob opened foo last, and did the last write, Alice was the 
last to close a transaction.  If you want to avoid this situation, you 
use locking.  Locking isn't mandatory, but neither are transactions.


But I could be entirely wrong also.


Re: reiser4 status (correction)

2006-07-21 Thread Mike Benoit
On Fri, 2006-07-21 at 16:06 -0500, David Masover wrote:
> Mike Benoit wrote:
> 
> > Tuning fsync will fix the last wart on Reiser4 as far as benchmarks are
> > concerned won't it? Right now Reiser4 looks excellent on the benchmarks
> > that don't use fsync often (mongo?), but last I recall the fsync
> > performance was so poor it overshadows the rest of the performance. It
> > would also probably be more useful to a much wider audience, especially
> > if Namesys decides to charge for the repacker.
> 
> If Namesys does decide to charge for the repacker, I'll have to consider 
> whether it's worth it to pay for it or to use XFS instead.  Reiser4 
> tends to become much more fragmented than most other Linux FSes -- 
> purely subjective, but probably true.
> 

I would like to see some actual data on this. I haven't used Reiser4 for
over a year, and when I did it was only to benchmark it. But Reiser4
allocates on flush, so in theory this should decrease fragmentation, not
increase it. Due to this I question what you are _really_ seeing, or if
perhaps it is a bug in the allocator? Why would XFS or any other
multi-purpose file system resist fragmentation noticeably more then
Reiser4 does.

I don't think the repacker is designed to be a "must have" for every
Reiser4 installation. If it was, I would consider Reiser4 to be
seriously flawed. Instead I think it is simply designed to improve
certain workloads that may cause high fragmentation in hopes of keeping
I/O speeds at their peek. 

Am I correct in this assumption Hans?
 
No Linux file system that I'm aware of has a defragmentor, but they DO
become fragmented, just not near as bad as FAT32 used to when MS created
their defragmentor. The highest "non-contiguous" percent I've seen with
EXT3 is about 12%, FAT32 I have seen over 50%, and NTFS over 30%. In
fact I'm running in to a fragmentation issue with ReiserV3 right now
that Jeff is working on, but it is more of a worst case scenario issue,
not a regular occurrence issue.

For "normal" workloads I doubt you would notice much difference at all
by using a repacker, 10% maybe? Which is one of the reasons you probably
haven't seen a repacker for EXT2/3, even though I'm sure it would
improve performance for some people.

-- 
Mike Benoit <[EMAIL PROTECTED]>


signature.asc
Description: This is a digitally signed message part


Re: somewhat OT query on journalling

2006-07-21 Thread Andreas Schäfer
> Don't get too excited -- the transactions probably aren't done yet. Without 
> those, no filesystem that claims to journal data is really any better than a 
> filesystem which only journals metadata.  Even once 
> they are implemented (or even if they are already), applications have to 
> support them directly.

Actually, I think transactions in a filesystem context are a bit
different from the transactions you know form databases. Generally
speaking, a transaction denotes a transition from one valid state to
another. This transition should either be performed completely or --
in case of errors -- performed not at all (a.k.a. "roll back").

Databases allow application defined transactions (i.e. the
application specifies when a valid state is being left and when one is
reached again).

IMHO for filesystems a transaction denotes the flushing of write
buffers. "metadata only" transactions/journaling mean that even after
a crash the file itself will be readable (and not pointing to
e.g. sector -4711 ...). "data _and_ metadata" now means that the
filesystem does also guarantee that the data itself is written
completely.

Please correct my if I got this totally wrong.

-Andreas



pgpmfUVRQiFUp.pgp
Description: PGP signature


Re: reiser4 status (correction)

2006-07-21 Thread David Masover

Mike Benoit wrote:


Tuning fsync will fix the last wart on Reiser4 as far as benchmarks are
concerned won't it? Right now Reiser4 looks excellent on the benchmarks
that don't use fsync often (mongo?), but last I recall the fsync
performance was so poor it overshadows the rest of the performance. It
would also probably be more useful to a much wider audience, especially
if Namesys decides to charge for the repacker.


If Namesys does decide to charge for the repacker, I'll have to consider 
whether it's worth it to pay for it or to use XFS instead.  Reiser4 
tends to become much more fragmented than most other Linux FSes -- 
purely subjective, but probably true.



ReiserV3 is used on a lot of mail and squid proxy servers that deal with
many small files, and these work loads usually call fsync often.

[...]

But neglecting fsync performance will just put a sour taste in their
mouth. 


So will neglecting fragmentation, only worse.  At least fsync is slow up 
front.  Fragmentation will be slow much farther in, when the mailserver 
has already been through one painful upgrade.  Charging for the repacker 
just makes it worse.



On top of that, I don't see how a repacker would help these work loads
much as the files usually have a high churn rate. Packing them would
probably be a net loss as the files would just be deleted in 24hrs and
replaced by new ones.


Depends.  Some will, some won't.  My IMAP server does have a lot of 
churning, but there's also the logs (which stay for at least a month or 
two before they rotate out), and since it's IMAP, I do leave quite a lot 
of files alone.


v3 is also used on a lot of web servers, at least where I used to work 
-- some areas will be changing quite a lot, and some areas not at all. 
Changing a lot means fragmentation will happen, not changing at all 
means repacking will help.


These issues may be helped by partitioning, if you know how you're going 
to split things up.  But then, how do you partition in the middle of a 
squid server?  A lot of people visit the same sites every day, checking 
for news, but that means plenty of logos, scripts, and other things 
won't change -- but plenty of news articles will change every couple hours.



Very few people will (or should) disable fsync as David suggests, I
don't see that as a solution at all, even if it is temporary.


I guess the temporary solution is to incur a pretty big performance hit. 
 But it comes back to, which is more of a performance problem, fsync or 
fragmentation?


And I really would like to hear a good counter-argument to the one I've 
given for disabling fsync.  But even if we assume fsync must stay, do we 
have any benchmarks on fragmentation versus fsync?


But maybe it's best to stop debating, since both will be done 
eventually, right?


Re: reiser4 status (correction)

2006-07-21 Thread Mike Benoit
On Fri, 2006-07-21 at 02:44 -0600, Hans Reiser wrote:
> fix fsync performance (est. 1 week of time to make post-commit writes
> asynchronous, maybe 3 weeks to create fixed-reserve for write twice
> blocks, and make all fsync blocks write twice)
> 
> write repacker (12 weeks).
> 
> I am not sure that putting the repacker after fsync is the right choice
> 

Tuning fsync will fix the last wart on Reiser4 as far as benchmarks are
concerned won't it? Right now Reiser4 looks excellent on the benchmarks
that don't use fsync often (mongo?), but last I recall the fsync
performance was so poor it overshadows the rest of the performance. It
would also probably be more useful to a much wider audience, especially
if Namesys decides to charge for the repacker.

ReiserV3 is used on a lot of mail and squid proxy servers that deal with
many small files, and these work loads usually call fsync often. My
guess is that ReiserV3 users are the most likely to migrate to Reiser4,
because they already know the benefits of using a "Reiser" file system.
But neglecting fsync performance will just put a sour taste in their
mouth. 

On top of that, I don't see how a repacker would help these work loads
much as the files usually have a high churn rate. Packing them would
probably be a net loss as the files would just be deleted in 24hrs and
replaced by new ones.

Very few people will (or should) disable fsync as David suggests, I
don't see that as a solution at all, even if it is temporary.

-- 
Mike Benoit <[EMAIL PROTECTED]>


signature.asc
Description: This is a digitally signed message part


Re: storing images thumbnails as pseudo files?

2006-07-21 Thread David Masover

Pysiak Satriani wrote:

Hello,

suppose pseudo files, file-as-directory are on my r4 partition and are usuable.

Does namesys' vision allow things like storing image thumbnails "inside the file
itself" ?


Ultimately, yes.  Currently, I don't know.

The most common use of pseudofiles, for instance, is to access metadata 
we already have.  For instance, Reiser4 stores file permissions the same 
way as anyone else, but you can access foo/..pseudo/rwx.


That's an example.  I haven't been in this community for a year or more, 
so I have no idea what the delimiter is.


For now, I don't think you can have arbitrary normal files within files, 
just metadata that has a plugin already.  You might try something like 
this, though:



$ mkdir -p .thumbs/test.jpg

// create 150px thumbnail
$ jpgtn -s 150 -H test.jpg

// move the thumbnail "into" the file
$ mv tn_test .thumbs/test.jpg/150px

// the same but a 100px thumbnail
$ jpgtn -s 100 -H test.jpg
$ mv tn_test .thumbs/test.jpg/100px


Ultimately, the Namesys vision would allow for something like this to 
happen:


$ cp /mnt/camera/test.jpg .

Then your link would look something like this:



The thumbnail would be generated by the FS (probably by calling jpgtn or 
a similar tool) the first time it was accessed, then cached until disk 
space is needed for something else.


The rest of the kernel development list seems to believe that such 
functionality would be better achieved by hacking glibc or Apache to 
emulate that, rather than having it be a feature of the filesystem itself.


Disclaimer:  I don't speak for Namesys, and I don't work here.  While 
I'm pretty confident I understand their vision, the final word on 
anything Reiser is always from Hans Reiser.


Re: Utilizing OSDL's STP for Reiser4 testing/benchmarks?

2006-07-21 Thread Hans Reiser
Elena, on Monday can you comment on this in detail?

Thanks,

Hans

Mike Benoit wrote:

>Has the Reiser4 team looked at utilizing the OSDL Scalable Test Platform
>(STP) service to benchmark and test Reiser4 patches? They seem to offer
>a wide variety of hardware to test on, and already have some file system
>benchmarks available to choose from. 
>
>I noticed the Namesys benchmark page hasn't been updated in quite some
>time, understandably so, as benchmarking can be incredibly time
>consuming.  Perhaps using the STP would make this much easier to do more
>frequently?
>
>It would be really nice to see how the batch_write/read(?) patches to
>the kernel affect things, and how Reiser4 performance progresses over
>time. I know Hans has mentioned CPU usage dropping in recent patches,
>but actual numbers from a benchmark are usually more compelling to see.
>
>User Guide:
>http://www.osdl.org/lab_activities/kernel_testing/stp/guide.html/document_view
>
>Test Details:
>http://www.osdl.org/lab_activities/kernel_testing/stp/test_details.html/document_view
>
>  
>



storing images thumbnails as pseudo files?

2006-07-21 Thread Pysiak Satriani
Hello,

suppose pseudo files, file-as-directory are on my r4 partition and are usuable.

Does namesys' vision allow things like storing image thumbnails "inside the file
itself" ?

Example using jpgtn (jpgtn creates thumbnails of jpg files)

// create 150px thumbnail
$ jpgtn -s 150 -H test.jpg

// move the thumbnail "into" the file
$ mv tn_test test.jpg/thumbs/150px

// the same but a 100px thumbnail
$ jpgtn -s 100 -H test.jpg
$ mv tn_test test.jpg/thumbs/100px

Then I could write HTML code like this:



This idea came to me because of a project that has to create many thumbnails
for many files. That requires to make a naming scheme, parse filenames and
don't make one mistake. If I could do things this way it would be much
easier and elegant to do.

But that's just an idea from someone not really knowledgible enough in
the area.

Best regards,
Maciej Soltysiak




Re: reiser4 status (correction)

2006-07-21 Thread David Masover

Hans Reiser wrote:


I am not sure that putting the repacker after fsync is the right choice


Does the repacker use fsync?  I wouldn't expect it to.

Does fsync benefit from a properly packed FS?  Probably.

Also, while I don't expect anyone else to be so bold, there is a way 
around fsync performance:  Disable it.  Patch the kernel so that any 
fsync call from userspace gets ignored, but lie and tell them it worked. 
 Basically:


 asmlinkage long sys_fsync(unsigned int fd)
 {
-   return do_fsync(fd, 0);
+   return 0;   // do_fsync(fd, 0);
 }

In order to make this sane, you should have backups and an Uninterrupted 
Power Supply.  In the case of a loss of power, the box should notice and 
immediately sync, then either shut down or software suspend.  Any UPS 
battery should be able to handle the amount of time it takes to shut the 
system off.


Since anything mission critical should have backups and a UPS anyway, 
the only problem left is what happens if the system crashes.  But system 
crashes are something you have to plan for anyway.  Disks fail -- stuff 
happens.  RAID won't save you -- the RAID controller itself will 
eventually fail.


So suppose you're running some very critical server -- for now, chances 
are it's running some sort of database.  In this case, what you really 
want is database replication.  Have at least two servers up and running, 
and consider the transaction complete not when it hits the disk, but 
when all running servers acknowledge the transaction.  The RAM of two 
boxes should be safer than the disk of one.


What about a repacker?  The best I can do to hack around that is to 
restore the whole box from backup every now and then, but this requires 
the box to be down for awhile -- it's a repacker, but not an online one. 
 In this case, the solution would be to have the same two servers 
(replicating databases), and bring first one down, and then the other.


That would make me much more nervous than disabling fsync, though, 
because now you only have the one server running, and if it goes down... 
 And depending on the size of the data in question, this may not be 
feasable.  It seems entirely possible that in some setups like this, the 
only backup you'd be able to afford would be in the form of replication.


In my own personal case, I'd prefer the repacker to tuning fsync.  But 
arguments could be made for both.


Re: somewhat OT query on journalling

2006-07-21 Thread David Masover

Payal Rathod wrote:

On Wed, Jul 19, 2006 at 01:28:38PM -0700, Hans Reiser wrote:

V3 defaults to metadata only, V4 does data also because we can do it
without performance loss.


wwow!!! 


Don't get too excited -- the transactions probably aren't done yet. 
Without those, no filesystem that claims to journal data is really any 
better than a filesystem which only journals metadata.  Even once they 
are implemented (or even if they are already), applications have to 
support them directly.


Regarding transactions in general, you should probably look for some 
papers or tutorials for how they are implemented in databases.  You 
might also read the Reiser4 whitepaper for an idea of how they could be 
implemented in a filesystem.


But no transactional system will work unless applications at least know 
about it.  This is why apps currently rely on features that are known to 
be atomic in a filesystem.  For instance, look at maildirs -- they are 
effectively 100% data journaled on any filesystem that journals metadata 
properly.



reiserfs does not use inodes at all.  see our website for more.


Any particular page you are referring to? I didn't see a page for that 
there.


Front page of Namesys.com has the Reiser4 whitepaper.  It's a bit out of 
date, as some features still listed for 4.0 have been pushed to 4.1 or 
later, but I don't think the overall plan has changed.


Re: 2.6.17 patch is in testing now

2006-07-21 Thread Ingo Bormuth
On 2006-07-18 13:43, Hans Reiser wrote:
> If testing goes well we will release it tomorrow.

Hmm, so what would you say - should it be regarded as released ?



-- 
Ingo Bormuth, voicebox & telefax: +49-12125-10226517   '(~o-o~)'
public key 86326EC9, http://ibormuth.efil.de/contact   --ooO--(.)--Ooo--


pgpBb5c6VVrTG.pgp
Description: PGP signature


Utilizing OSDL's STP for Reiser4 testing/benchmarks?

2006-07-21 Thread Mike Benoit
Has the Reiser4 team looked at utilizing the OSDL Scalable Test Platform
(STP) service to benchmark and test Reiser4 patches? They seem to offer
a wide variety of hardware to test on, and already have some file system
benchmarks available to choose from. 

I noticed the Namesys benchmark page hasn't been updated in quite some
time, understandably so, as benchmarking can be incredibly time
consuming.  Perhaps using the STP would make this much easier to do more
frequently?

It would be really nice to see how the batch_write/read(?) patches to
the kernel affect things, and how Reiser4 performance progresses over
time. I know Hans has mentioned CPU usage dropping in recent patches,
but actual numbers from a benchmark are usually more compelling to see.

User Guide:
http://www.osdl.org/lab_activities/kernel_testing/stp/guide.html/document_view

Test Details:
http://www.osdl.org/lab_activities/kernel_testing/stp/test_details.html/document_view

-- 
Mike Benoit <[EMAIL PROTECTED]>


signature.asc
Description: This is a digitally signed message part


Re: create very large file system

2006-07-21 Thread Stephen Tweedie
Hi,

On Wed, Jul 19, 2006 at 07:10:56AM -0500, Mark F wrote:
> Suse Linux Enterprise Server 9 SP3
> 
> I've tried to create a large 5TB file system using both reiserfs and ext3 
> and both have failed.
> 
> I end up with only a 1.5TB file system.  Does anyone know why this doesn't 
> work, what to do to fix it?

I fixed a bug in mke2fs that had this result over a year ago, so a
recent e2fsprogs should fix it.

Failing that, there's a workaround: use "mke2fs -b 4096" to prevent
mke2fs from trying to work out the device size in units of 1k blocks.
Counting in 4k blocks prevents a 32-bit overflow.

--Stephen


Re: fsck.reiserfs --rebuild-tree out of disk space aborted

2006-07-21 Thread Vladimir V. Saveliev
hello

On Fri, 2006-07-21 at 10:09 +1000, Joel Heenan wrote:
> Hi,
> 
> I can home about two weeks ago and found my media box locked up. I was
> able to discover that it had filled up its /dev/md2 partition (mounted
> on /home) which surprised me because it is 550 gigs. Perhaps mythtv
> went nuts and used it all up. It was only a temporary thing I was
> going to move mythtv to another partition anyway and boy I wish I had
> now :-P.
> 
> Anyway I rebooted and the fsck said I had to run rebuild-tree. So I
> ran that and a few days later it said "out of disk space, aborted". I
> can't mount the partition it says "No folder found I believe". I tried
> it a few times with both the reiserfsprogs from etch:
> 
> ii  reiserfsprogs3.6.19-2
> User-level tools for ReiserFS filesystems
> 
> and the latest ones downloaded from the website (3.6.19). I am
> currently trying it with the -S option.
> 
there is no need to use -S in your case.

> I'm running a custom 2.6.12 kernel which is basically the same as the
> default debian one except it includes some drivers for my dvico fusion
> tv tuner.
> 
> I read that the best way to fix this problem is to dd to a bigger
> partition but there is really no easy way for me to do that - it will
> probably involve me purchasing 2x300gig partitions, raiding them, then
> performing the restore.
> 
> Please let me know if there is a way I can fix this without going to a
> bigger partition.
> 
> Here is output of the fsck:
> 
> """
> fsck.reiserfs --rebuild-tree /dev/md2
> reiserfsck 3.6.19 (2003 www.namesys.com)
> 
> *
> ** Do not  run  the  program  with  --rebuild-tree  unless **
> ** something is broken and MAKE A BACKUP  before using it. **
> ** If you have bad sectors on a drive  it is usually a bad **
> ** idea to continue using it. Then you probably should get **
> ** a working hard drive, copy the file system from the bad **
> ** drive  to the good one -- dd_rescue is  a good tool for **
> ** that -- and only then run this program. **
> ** If you are using the latest reiserfsprogs and  it fails **
> ** please  email bug reports to reiserfs-list@namesys.com, **
> ** providing  as  much  information  as  possible --  your **
> ** hardware,  kernel,  patches,  settings,  all reiserfsck **
> ** messages  (including version),  the reiserfsck logfile, **
> ** check  the  syslog file  for  any  related information. **
> ** If you would like advice on using this program, support **
> ** is available  for $25 at  www.namesys.com/support.html. **
> *
> 
> Will rebuild the filesystem (/dev/md2) tree
> Will put log info to 'stdout'
> 
> Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes
> Replaying journal..
> Reiserfs journal '/dev/md2' in blocks [18..8211]: 0 transactions replayed
> ###
> reiserfsck --rebuild-tree started at Mon Jul 17 21:41:44 2006
> ###
> 
> Pass 0:
> ### Pass 0 ###
> Loading on-disk bitmap .. ok, 116676032 blocks marked used
> Skipping 11771 blocks (super block, journal, bitmaps) 116664261 blocks
> will be read
> 0%20%block 85875092: The number of items (256) is incorrect,
> should be (1) - corrected
> block 85875092: The free space (1) is incorrect, should be (208) - corrected
> pass0: vpf-10110: block 85875092, item (0): Unknown item type found
> [33488896 65537 0x100 ??? (15)] - deleted
>left 0, 15065 /seccsec
> 108413 directory entries were hashed with "r5" hash.
> "r5" hash is selected
> Flushing..finished
> Read blocks (but not data blocks) 116664261
> Leaves among those 87768646
> - leaves all contents of which could not be
> saved and deleted 1
> Objectids found 108417
> 
> Pass 1 (will try to insert 87768645 leaves):
> ### Pass 1 ###
> Looking for allocable blocks .. finished
> 0%20%40%..Not enough allocable blocks, checking bitmap...there
> are 1 allocable blocks, btw
> 

I am working on a fix for this problem. But the above output looks
suspicious. 

What kind of files are there, would you describe?
Are they long? Did you mount reiserfs with any options?
Can you give me ssh access to the filesystem so that I could take a look
at it myself?


> out of disk space
> Aborted
> """
> 



Re: somewhat OT query on journalling

2006-07-21 Thread Payal Rathod
On Wed, Jul 19, 2006 at 01:28:38PM -0700, Hans Reiser wrote:
> V3 defaults to metadata only, V4 does data also because we can do it
> without performance loss.

wwow!!! 

> 
> >Also, is it true that now-a-days there is no such thing as inode "block"
> >since  for faster access the inodes are kept near the data itself?
> >  
> >
> reiserfs does not use inodes at all.  see our website for more.

Any particular page you are referring to? I didn't see a page for that 
there.

With warm regards,
-Payal


Re: reiser4 status (correction)

2006-07-21 Thread Sarath Menon

> Sigh, no, the repacker will probably be after 4.1
> The list of tasks for zam looks something like:
> fix bugs that arise
> debug read optimization code (CPU reduction only, has no effect on IO),
> 1 week est.  (would be nice if it was less)
> review compression code 1 day per week until it ships.
> fix fsync performance (est. 1 week of time to make post-commit writes
> asynchronous, maybe 3 weeks to create fixed-reserve for write twice
> blocks, and make all fsync blocks write twice)
> write repacker (12 weeks).

Well, this is free software, and is not backed u with a million dollars of 
funding. Trust me, these guys are doing a great job with the options they 
have. 

This is not a fliambait, but I personally feel that this is way too great 
considering the resources that the entire team has. I am more than willing to 
wait even though I have had occasional bad experiences with reiser{3,4}. 

The bottemline is that all versions of reiserfs have had great performance, 
even though with its own hiccups (which fs doesn't have ??) and simply put, 
it is a fantastic peice of code.



-- 
My mother loved children -- she would have given anything if I had been one.
-- Groucho Marx


Re: reiser4 status (correction)

2006-07-21 Thread Hans Reiser
David Masover wrote:

> Hans Reiser wrote:
>
>> On a more positive note, Reiser4.1 is getting closer to release
>
>
> Good news!  But it's been awhile since I've followed development, and
> the homepage seems out of date (as usual).  Where can I find a list of
> changes since v4?
>
> By "out of date", I mean things like this:
>
> "Reiser4.1 will modify the repacker to insert controlled "air holes",
> as it is well known that insertion efficiency is harmed by overly
> tight packing."

Sigh, no, the repacker will probably be after 4.1

The list of tasks for zam looks something like:

fix bugs that arise

debug read optimization code (CPU reduction only, has no effect on IO),
1 week est.  (would be nice if it was less)

review compression code 1 day per week until it ships.

fix fsync performance (est. 1 week of time to make post-commit writes
asynchronous, maybe 3 weeks to create fixed-reserve for write twice
blocks, and make all fsync blocks write twice)

write repacker (12 weeks).

I am not sure that putting the repacker after fsync is the right choice

The task list for vs looks like:

* fix bugs as they arise.

* fix whatever lkml complains about that either seems reasonable, or
that akpm agrees with.

* Help edward get the compression plugins out the door.

* Improve fsck's time performance.

* Fix any V3 bugs that Chris and Jeff don't fix for us.  Which reminds
me, I need to check on whether the 90% full bug got fixed



Re: fsck.reiserfs --rebuild-tree out of disk space aborted

2006-07-21 Thread Hans Reiser
vs will try to help you  Did Jeff fix the mythtv performance problem
yet?  If not, vs, please rip out the "optimization" which goes looking
for the perfect length spot for files, and send both joel and akpm the
patch.  It is really not such a bad algorithm to just use the spots that
are free in the order that they are found, and it can certainly be used
until a well tested and benchmarked alternative algorithm is presented
to me in detail.

Hans

Joel Heenan wrote:

> Hi,
>
> I can home about two weeks ago and found my media box locked up. I was
> able to discover that it had filled up its /dev/md2 partition (mounted
> on /home) which surprised me because it is 550 gigs. Perhaps mythtv
> went nuts and used it all up. It was only a temporary thing I was
> going to move mythtv to another partition anyway and boy I wish I had
> now :-P.
>
> Anyway I rebooted and the fsck said I had to run rebuild-tree. So I
> ran that and a few days later it said "out of disk space, aborted". I
> can't mount the partition it says "No folder found I believe". I tried
> it a few times with both the reiserfsprogs from etch:
>
> ii  reiserfsprogs3.6.19-2
> User-level tools for ReiserFS filesystems
>
> and the latest ones downloaded from the website (3.6.19). I am
> currently trying it with the -S option.
>
> I'm running a custom 2.6.12 kernel which is basically the same as the
> default debian one except it includes some drivers for my dvico fusion
> tv tuner.
>
> I read that the best way to fix this problem is to dd to a bigger
> partition but there is really no easy way for me to do that - it will
> probably involve me purchasing 2x300gig partitions, raiding them, then
> performing the restore.
>
> Please let me know if there is a way I can fix this without going to a
> bigger partition.
>
> Here is output of the fsck:
>
> """
> fsck.reiserfs --rebuild-tree /dev/md2
> reiserfsck 3.6.19 (2003 www.namesys.com)
>
> *
> ** Do not  run  the  program  with  --rebuild-tree  unless **
> ** something is broken and MAKE A BACKUP  before using it. **
> ** If you have bad sectors on a drive  it is usually a bad **
> ** idea to continue using it. Then you probably should get **
> ** a working hard drive, copy the file system from the bad **
> ** drive  to the good one -- dd_rescue is  a good tool for **
> ** that -- and only then run this program. **
> ** If you are using the latest reiserfsprogs and  it fails **
> ** please  email bug reports to reiserfs-list@namesys.com, **
> ** providing  as  much  information  as  possible --  your **
> ** hardware,  kernel,  patches,  settings,  all reiserfsck **
> ** messages  (including version),  the reiserfsck logfile, **
> ** check  the  syslog file  for  any  related information. **
> ** If you would like advice on using this program, support **
> ** is available  for $25 at  www.namesys.com/support.html. **
> *
>
> Will rebuild the filesystem (/dev/md2) tree
> Will put log info to 'stdout'
>
> Do you want to run this program?[N/Yes] (note need to type Yes if you
> do):Yes
> Replaying journal..
> Reiserfs journal '/dev/md2' in blocks [18..8211]: 0 transactions replayed
> ###
> reiserfsck --rebuild-tree started at Mon Jul 17 21:41:44 2006
> ###
>
> Pass 0:
> ### Pass 0 ###
> Loading on-disk bitmap .. ok, 116676032 blocks marked used
> Skipping 11771 blocks (super block, journal, bitmaps) 116664261 blocks
> will be read
> 0%20%block 85875092: The number of items (256) is incorrect,
> should be (1) - corrected
> block 85875092: The free space (1) is incorrect, should be (208) -
> corrected
> pass0: vpf-10110: block 85875092, item (0): Unknown item type found
> [33488896 65537 0x100 ??? (15)] - deleted
>   left 0, 15065
> /seccsec
> 108413 directory entries were hashed with "r5" hash.
>"r5" hash is selected
> Flushing..finished
>Read blocks (but not data blocks) 116664261
>Leaves among those 87768646
>- leaves all contents of which could not be
> saved and deleted 1
>Objectids found 108417
>
> Pass 1 (will try to insert 87768645 leaves):
> ### Pass 1 ###
> Looking for allocable blocks .. finished
> 0%20%40%..Not enough allocable blocks, checking bitmap...there
> are 1 allocable blocks, btw
>
> out of disk space
> Aborted
> """
>
>




Re: other system with datacorruption (2.425 + datalogging patches)

2006-07-21 Thread Francisco Javier Cabello
Hello,
The rebuild tree succeed. I send you the output.

Regards,

Paco

On Thursday, 20 de July de 2006 14:52, Francisco Javier Cabello wrote:
> Hello,
> I have other system with data corruption.
> I send you the output of 'reiserfsck --check'
>
> Regards,
>
> Paco

-- 
One of my most productive days was throwing away 1000 lines of code (Ken 
Thompson)
-
PGP fingerprint: AF69 62B4 97EB F5BB 2C60  B802 568A E122 BBBE 5820
PGP Key available at http://pgp.mit.edu
-



reiserfsc_rebuild_tree.gz
Description: GNU Zip compressed data


pgpH54kj08TEI.pgp
Description: PGP signature