Re: Linux 2.4 - 2.6 migration
Did you enable IDE TCQ in your 2.6 kernel? To quote http://www.namesys.com/ download.html Don't enable TCQ (Tagged Command Queuing) in 2.5 (and modified 2.4) kernels. TCQ is known broken currently and it will corrupt your filesystems for sure. Quinn On Friday 07 November 2003 06:39 am, Sebastian Kaps wrote: Hi! Is there something concerning ReiserFS I should know when migrating from Linux 2.4 to Linux 2.6? I'm asking because a few days ago I downloaded and compiled 2.6.0-test9 just for curiosity. It booted fine and I went back to my standard kernel, which was 2.4.23-pre5 at that time. Right after booting the 2.4.x kernel, I got lots of the following messages in my logs: , | kernel: ide2(33,3):vs-4080: reiserfs_free_block: free_block | (2103:463529)[dev:blocknr]: bit already cleared ` The block number always differed. Gladly, reiserfsck --rebuild-tree helped to cure that, but there were lots of unusable files after that (e.g. parts of files overwritten with parts of other files). Is this suppose to happen? What caused this error? Another thing: I had to hard reboot my system a few times in the last few months. I also experienced that sometimes parts of files were overwritten with 0x00s or parts of other files. Can someone explain to me how this can happen? -- Ciao, Sebastian
Relocating files for faster boot/start-up on reiser(fs/4)
patterns and better allocation of the data. I hope to rerun these tests with Reiser4 (maybe 4.1) on the same hardware. I expect with a fresh (not fragmented) Reiser4 partition, the improvements will be more pronounced. But a repacker should allow more reproducible results and nearly perfect data placement for boot and app start-up. I hope with reiser4, relocation, and the new upstart (Ubuntu's sysvinit replacement) with good scripts, I will get this system to boot to usable in 30 seconds. And slowoffice (aka openoffice) to load in 6s cold. Am I overoptimistic? What about a mechanism to explicitly set or hint at item keys? Maybe someday, linux packages could include preferred file order information that a file system like reiser4 could use to order the files on disk resulting in fast load times without the need for the user to profile the app. I think there is a lot to be said for measuring access patterns and using that to set keys in addition to deducing it from semantics using a fibration plug-in. Thoughts? -- Quinn Harris
Re: Relocating files for faster boot/start-up on reiser(fs/4)
Peter, I think you misunderstood what and why I was doing this. Let me try to clarify. My test is far from perfect. Its mearly an exercise to verify the basic idea. Just by copying you are allowing reiser to optimize the dir. Exactly, but I am copying in a way that implicitly suggests what order those files will be accessed in. I was attempting to reorder the data on disk to minimize disk seeks with knowledge of the order that data will be accessed. This was done by taking advantage of the way reiser assigns keys to files based on their name and its affinity to match key order with block order. You're trying to duplicate what a tree-based design does automatically. This works because of the tree-based design of reiser. The reiser must assign each file (item actually) some key, why not take advantage of knowledge of the order those items will be accessed in? The current key assignment algorithm is a best guess at that given the limited information it has (file/directory name). Remember key assignment roughly translates to on disk position. The relocate script can leave the file system in the exact same state from a semantic standpoint (what files and directories are there) but relocate the data on disk. Copying those files to single directory with numeric names was a kludge to implicitly tell the file system to place those files in a specific order and near each other on disk. The rename step is to switch the old unoptimized file position with the new more optimized position. Moreover, remember that reiser packs files into clusters so that you may read more than just your one file from time to time which could end up adding time to your test. The boot optimization was over 3885 files. Ideally those files would be ordered head to tail in a sequence that perfectly matches the order they will be read. As a result multiple items in a node will all need to be read at nearly the same time. That didn't happen in my test, but it was much closer to that after I ran the relocate script than before. Hence the performance improvement. With this script, reiser4 and a repacker I have reason to believe the ordering will be nearly perfect. Of course, that is excluding random access patterns inside the same file and the directory data needed to get at the files. This basic technique can be made into a boot script much like the readahead script already in Ubuntu, just improved. Boot once with a profile option, it measures read patterns (already does this), then reorders data on disk with this trick, or maybe something better. Then the next time you boot its 1.5-2x faster. Better yet, including this profile information in the distro packages. When a package is installed this info is used to help assign item keys resulting in a better disk layout and faster boot times and no weird file copy rename mumbo jumbo. I bring this up here because I expect with reiser4, a repacker, and this trick, reiser4 could deliver at least 50% better reproducible real world boot and app load performance than any other file system. At least until other file system implement something similar, like what MS did with XP. Can something similar be done (or has been) on ext(2/3/4), XFS, JFS or other linux file systems? Windows XP boots much faster than Windows 2000 in part because it does what I am talking about. File access is recorded at boot, then the disk is defraged with this knowledge. Check out http://msdn.microsoft.com/msdnmag/issues/01/12/xpkernel/default.aspx under Prefetch. Also look at http://kerneltrap.org/node/2157 MS's implementation required implementing a defrag utility with a specific feature that could position disk data based on access logs. Reiser4 can do the same thing as part of its basic functionality with the addition of a much much simpler tool to help assign keys based on that access log. Then a repacker (when it devaporizes) can further optimize for that access pattern without any code specific to that purpose. Seems like good orthogonal design to me. Hope that clarifies. Like my previous post, whatever it did, it did it in way to many words. On Wednesday 13 September 2006 15:10, Peter wrote: On Wed, 13 Sep 2006 14:51:39 -0600, Quinn Harris wrote: Thoughts? Yes. Why on earth would you do this? By copying the files and renaming and hardlinking them is nothing a sysadmin would ever do. Just by copying you are allowing reiser to optimize the dir. You're trying to duplicate what a tree-based design does automatically. Moreover, remember that reiser packs files into clusters so that you may read more than just your one file from time to time which could end up adding time to your test. If reiser needs speedup it certainly won't be done by renaming files! JM$0.02 -- Quinn Harris
Re: Relocating files for faster boot/start-up on reiser(fs/4)
On Thursday 14 September 2006 23:15, Toby Thain wrote: On 14-Sep-06, at 6:23 PM, David Masover wrote: Quinn Harris wrote: On Thursday 14 September 2006 13:55, David Masover wrote: ... That is a good point. Recording the disk layout before and after to compare relative fragmentation would be a good idea. As well as randomizing the sequence as a sanity check. Also note that during boot I was using readahead on all 3885 files. So the kernel has a good opportunity to rearrange the reads. And the read sequence doesn't necessary match the order its needed (though I tried to get that). Speaking of which, did you parallize the boot process at all? Just off the top of my head, wouldn't that make the access sequence asynchronous thereby less predictable? (Although I'm sure it's a net win.) It could, but the kernel will try to reorder the outstanding block requests to reduce seek. If that is an overall win I don't know. In addition early in the boot, readahead-list or similar will tell the kernel to start reading most of the files need for the complete boot so they are already in memory when needed. Ubuntu does the readahead now and all my tests where with readahead. I'd estimate my system easily spent more than 50% of its boot time not touching the disk at all before I did that. Gentoo can do this, I'm not sure what else, as it kind of needs your init system to understand dependencies. ... My first test turned out to be on a heavily fragmented file system. I reinstalled Ubuntu Dapper with a fresh reiserfs file system and it booted in 1:07 (grub to desktop background appearing). After extending the time readahead-watch monitors files and running the reallocate script it now boots in 0:50. I wrote a little python script that uses the FIBMAP ioctl to check the blocks the files are using. From this I know the relocate script on this fresh file system is doing exactly what it was intended to do. I am also able to estimate how much it will improve performance by comparing the fragmentation before and after its run. I have learned that the delays on disk io for Ubuntu boot are dominated by rotational latency and not head seeks. The current readahead implementation orders the files by on disk location, substantially mitigating head seek time. But the latency is can easily double the time needed to load the same data. Subjectively (and objectively by about 6s) relocation and extending readahead-watch substantially improved Gnome boot and initial responsiveness. But, I need to measure how much of this was caused by just extending how much is read ahead vs. the reallocation. The current Ubuntu boot waits for hardware probing, DHCP and other things giving the disk readahead a chance to work. I think this reallocation might help a parallel boot more as the data will be needed sooner. So I changed my mind, I think parallel boot will highlight the reallocate advantage. Now I just need to test the hypothesis. Not sure if I would be better of trying initng or waiting for upstart (Ubuntus new init) to get scripts that actually parallel boot. The code for upstart is very clean and it has the backing of a major distro, so I have high hopes. Much like before, I was able to improve a 16.5s oowriter cold start to 14s with this reallocate script, with a cold start of 4.8s (OO 2.0.2, was using 2.0.3 before). It is evident to me that the readahead-watch is missing something on Open Office startup. It seems very possible to get OO to cold start in under 8s with the uses of reallocation and readahead right when it starts. My current scripts are at http://www.quinnh.org/reallocate.py (27 line reallocate script, expects dir /tmp/refrag to exist and takes the readahead-watch log as a paramater) http://www.quinnh.org/measure.py (uses FIBMAP to estimate the time needed to load the files in the passed readahead-watch log, uses average seek and and latency for estimate) http://www.quinnh.org/readahead-watch-time-order.patch (Patch against Ubuntu readahead-watch to add an order by access time option.) I will try to write a nice unified script that will profile, reallocate and do readahead for an application to speed it up. e.g. # reallocate.py oowriter. Run it once to profile and reallocate. drop_caches, Run it again and oowriter loads faster. I think Python will be the best language for this because its become relatively universal and its easy to understand for the uninitiated. This really isn't black magic so transparency is good. I personally prefer Ruby though.
Re: v3 rebuild-tree left system in unusable state because of space shortage
Are you still able to mount the fs read write? If so, you might see if you
can find files to delete, or possibly the 4G file was hardlinked to somewhere
else. This should list all files on the partition with the largest last.
find -type f -exec du {} \; | sort -n
Then wait a long time.
Does that turn up anything interesting?
KDE has a nice file size view graphic tool as a kpart for konqueror.
On Friday 15 September 2006 11:06, Grzegorz Jaśkiewicz wrote:
On 9/15/06, Vladimir V. Saveliev [EMAIL PROTECTED] wrote:
Could it be that the file was opened when you deleted it?
nope, I know that issue. I rebooted the system twice before trying any
other way.
Re: v3 rebuild-tree left system in unusable state because of space shortage
Scratch that last command use find -type f -printf %k\t%p\n | sort -n | tail -n 100 its much faster. On Friday 15 September 2006 11:06, Grzegorz Jaśkiewicz wrote: On 9/15/06, Vladimir V. Saveliev [EMAIL PROTECTED] wrote: Could it be that the file was opened when you deleted it? nope, I know that issue. I rebooted the system twice before trying any other way.
Re: reiser4: data recovery after mkfs.reiser4?
I really doubt there is any solution that would take less than a few hours. I am sure it is possible to recover much of the data but to the best of my knowledge no tool exists that can recover from an abandoned root node (for reiser4). Though I believe recovery in this case would just involve finding the root node (think that is reasonably tractable, but slow) fixing the superblock to point to that and let fsck do its thing. I don't think the root node has a magic number that advertises root, but internal nodes do have a recognizable signature and in principal one could deduce which is the root from a collection of the internal nodes. Note that reiser4 packs lots of data in single nodes. If you create a fresh fs with only a few small files they will reside entirely in the root node, which will be clobbered by a mkfs. There is a very good change that mkfs will clobber a little bit of data, but less than 4K. I am sure a few thousand dollars would buy you a solution. Maybe less. On Tuesday 19 December 2006 1:20 pm, Christian Trefzer wrote: Hi folks, a buggy script I wrote dared to mkfs a reiser4 partition after failing to properly tar up its contents - not that my life would depend on them, but it would save me a few hours if I could get the stuff back. Other than mkfs.reiser4, mount and unmount, nothing was done to the device ; ) Is there any way to recover most of what was stored on the now nuked fs? TIA, Chris
Re: Hans selling Namesys
All of the Reiser4 code as it exists today will always be available under GPL no matter who buys the copyright. The only reason I can see to buy Namesys is either to offer or use a re-licensed copy of the code or to take advantage of the knowledge of the current Namesys employees. There might be some value in existing contracts with DARPA or whoever but I am not sure. The re-license thing could be very difficult if there are many people who decide on if it will be re-licensed. This is why Hans has always retained exclusive copyright to all the code. I would question if re-licensing would be feasible with any structure that could be called community ownership. In the short term I believe the current employees of Namesys (how many are there now?) are the most valuable asset. It takes time to become acclimated to such a complex piece of code. Having to rehire a new set of people would easily set the work back by a few months to a year. On the other hand, I also believe that exclusive rights to the code could be very valuable to companies like Apple or even everybodies favorite, Microsoft. Even with knowledge of the mechanisms Reiser4 uses, I expect it could cost another company over a million dollars more to completely re-implement a similar file system compared to having uninhibited use of the existing code. I could see a company like Apple using a file system like Reiser4 to improve the performance of the existing file system and more importantly to enable acceptable performance for a namespace the unifies the file system and XML (chosen because of its widespread use) allowing for the elimination of the save file paradigm. Basically, applications could store data in an in memory XML DOM (as I expect Open Office does today) which would be seamlessly written to disk after each transaction on that DOM. (Reiser4 can do this much better than any other FS because of efficient storage of small items and the reiser syscall if ever finished). Such a system could claim, Never loose data, Ever. (This is what databases have been doing for decades.) Actually you could probably loose a few seconds of data as I expect flushing to disk after each character typed wouldn't be acceptable. Though even this problem could be eliminated with an small NVRAM, Flash and HD hybrid. Wrap the NVRAM and flash in hard casing, mix in a good real time internet backup scheme (plugins and efficient extensible metadata would help here), and it would become possible to insure against data loss at a reasonable cost. Something like, with this laptop and OS, if you loose any data, even if you drop it off a 20 story building 1 second after typing that last word, we will cut you a check for $100,000. Some limitations apply. I think many would pay an extra few hundred dollars for something like that. Of course, getting to that place would require alot of further work but Reiser4 would be a good start. I would rather see this under GPL in Linux but this verbage was to elaborate on one reason that the existing code would be worth a substantial chunk of change. This could also be a reason why it would be in the interest of existing Linux companies to ensure development continues. Bottom line, Han's best hope at making any significant amount on this sale would rest in a company seeing the value in Reiser4 wishing to re-license the code. On Monday 25 December 2006 4:22 pm, [EMAIL PROTECTED] wrote: I'm an offer to buy some or little part of the legal properties of Namesys, still compromised with open source community. How much it costs at all? I work with reiser4 in my own enterprise private distribution and want this project lasting for a long time. André Zaiats. Codix - São Paulo - Brazil Hello, from reading all this stuff it seems to look like a competitors intrigue. But we don't have any evidences upon this neither do we have a sentence, yet... Can we build up a ReiserFS-community based fund to buy or save Namesys over time to help continue the current development? Just an idea. From my point of view the there are three general alternatives: * let/help someone really interested to buy Namesys * increase Hans' financial input directly to help himReiserFS * ReiserFS34 get bought by someone and become commercial or will vanish at once As I followed ReiserFS since 1997(?+-) but really don't know enough about the current case I'm not sure with my priorities in the list above. Best wishes to all contributors, my best wishes for your Xmas! Manuel Joe Feise wrote on '06-12-25 20:55: http://www.wired.com/news/technology/0,72342-0.html
