[ntfs-3g-devel] [PATCH] compress.c: Speed up NTFS compression algorithm

2014-08-09 Thread Eric Biggers
@@ * Copyright (c) 2004-2006 Szabolcs Szakacsits * Copyright (c) 2005 Yura Pakhuchiy * Copyright (c) 2009-2013 Jean-Pierre Andre + * Copyright (c) 2014 Eric Biggers * * This program/include file is free software; you can redistribute it and/or * modify it under the terms

Re: [ntfs-3g-devel] [PATCH] compress.c: Speed up NTFS compression algorithm

2014-08-09 Thread Eric Biggers
Also, results from tests I did copying a file to compressed directory on a NTFS-3g mount, with time elapsed and compressed sizes shown: silesia_corpus.tar (211,957,760 bytes) Current 43.318s 111,230,976 bytes Proposed12.903s 111,751,168 bytes canterbury_corpus.tar

Re: [ntfs-3g-devel] [PATCH] compress.c: Speed up NTFS compression algorithm

2014-08-10 Thread Eric Biggers
On Sun, Aug 10, 2014 at 10:45:16AM +0200, Jean-Pierre André wrote: You have defined the hash table on static data, and I do not want to enter into the meanings of static data in shared objects in various operating systems (allowed or not, shared by threads or not...). I prefer to have it

Re: [ntfs-3g-devel] [PATCH] compress.c: Speed up NTFS compression algorithm

2014-08-10 Thread Eric Biggers
On Sun, Aug 10, 2014 at 11:18:49AM +0200, Jean-Pierre André wrote: Hi, Did you compare with the Microsoft implementation ? I have only checked the biggest file in IE7 update for WinXP (WINDOWS/ie7updates/KB963027-IE7/ieframe.dll) with cluster size 4096 : Original size 6066688

Re: [ntfs-3g-devel] [PATCH] compress.c: Speed up NTFS compression algorithm

2014-08-10 Thread Eric Biggers
On Sun, Aug 10, 2014 at 05:29:53PM +0200, Jean-Pierre André wrote: For a better way you would have to identify which is the dll which compresses , and submit compression tasks with some control over the durations. RtlCompressBuffer() in ntdll.dll can do LZNT1 compression, which I think is

Re: [ntfs-3g-devel] [PATCH] compress.c: Speed up NTFS compression algorithm

2014-08-10 Thread Eric Biggers
Yura Pakhuchiy * Copyright (c) 2009-2014 Jean-Pierre Andre + * Copyright (c) 2014 Eric Biggers * * This program/include file is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License as published @@ -21,17 +22,6 @@ * along

Re: [ntfs-3g-devel] [PATCH] compress.c: Speed up NTFS compression algorithm

2014-08-10 Thread Eric Biggers
) 2014 Eric Biggers * * This program/include file is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License as published @@ -21,17 +22,6 @@ * along with this program (in the main directory of the NTFS-3G * distribution in the file

[ntfs-3g-devel] Multiple unnamed data streams?

2015-08-17 Thread Eric Biggers
Hi, I had a report of a file's data disappearing when an NTFS volume was archived using wimlib, which uses libntfs-3g to read from NTFS volumes. What seems to have happened is that libntfs-3g reported two unnamed data streams for a file: one nonempty and one empty, and wimlib happened to store

Re: [ntfs-3g-devel] Experimental support for Windows 10 System Compressed files

2015-07-22 Thread Eric Biggers
on Windows 10 ? Glancing at your code, I do not see anything related to (sparse) cluster allocation. Does that mean these files are not seekable and must be read/written sequentially ? Regards Jean-Pierre Eric Biggers wrote: Hello, I've made an experimental fork of ntfs-3g

[ntfs-3g-devel] Experimental support for Windows 10 System Compressed files

2015-07-15 Thread Eric Biggers
Hello, I've made an experimental fork of ntfs-3g that supports reading the System Compressed files that are / will be supported by Windows 10. This feature allows rarely-modified files to be stored using XPRESS or LZX compression, with stronger compression than the LZNT1 compression built into

Re: [ntfs-3g-devel] [BUG] ntfs_attr_pwrite() may return short count when writing to compressed attribute

2015-10-31 Thread Eric Biggers
in practice. What if the existing ntfs_attr_pwrite() was simply moved to an internal function, and ntfs_attr_pwrite() was written as a retry loop around the internal function? Eric On Sat, Oct 31, 2015 at 07:06:01PM +0100, Jean-Pierre André wrote: > Hi Eric, > > Eric Biggers wro

Re: [ntfs-3g-devel] ENOSPC when adding file to directory with near-full MFT record

2015-11-05 Thread Eric Biggers
On Wed, Nov 04, 2015 at 08:10:56AM +0100, Jean-Pierre André wrote: > Hi Eric, > > Attached is the patch (simpler than I first thought). > > Jean-Pierre Thanks. I tested the patch and it made the ENOSPC problem go away. I'm currently trying to track down a corruption problem that seems to

[ntfs-3g-devel] [PATCH] acls.c: fix validation of SID subauthority count

2015-07-12 Thread Eric Biggers
ntfs_valid_sid() required that the subauthority count be between 1 and 8 inclusively. However, Windows permits more than 8 subauthorities as well as 0 subauthorities: - The install.wim file for the latest Windows 10 build contains a file whose DACL contains a SID with 10 subauthorities.

[ntfs-3g-devel] [RESEND] Incorrect handling of attribute starting with single-cluster extent

2015-11-14 Thread Eric Biggers
[Resending with only the script attached, since the original apparently didn't go through] Hi, I finally have more information, and a potential solution, for the apparent NTFS corruption bug I've been encountering during randomized tests. The bug, as I've been experiencing it, results in an

Re: [ntfs-3g-devel] Experimental support for Windows 10 "System Compressed" files

2015-09-22 Thread Eric Biggers
Hi, "WOF compression" is as good as the other names. It still seems slightly wrong because WOF (the "Windows Overlay Filesystem Filter") is a more general feature, and this is actually the *second* compression technology that Microsoft has built on top of it (the first was "WIMBoot"). For now,

Re: [ntfs-3g-devel] Experimental support for Windows 10 "System Compressed" files

2015-12-05 Thread Eric Biggers
On Fri, Dec 04, 2015 at 10:03:41AM +0100, Jean-Pierre André wrote: > For creating a new compressed file, the procedure would be : > - create a new void file > - "truncate" it to the desired size (hence a void sparse file) > - set reparse data for the desired compression mode > - feed the data

Re: [ntfs-3g-devel] Experimental support for Windows 10 "System Compressed" files

2015-12-05 Thread Eric Biggers
Hi, On Fri, Dec 04, 2015 at 10:03:41AM +0100, Jean-Pierre André wrote: > Hi Eric, > > Please see http://jp-andre.pagesperso-orange.fr/systcomp.tar.gz > with (most of) your comments taken into account. It generally looks good. I did a basic test of the system compression plugin (reading files

Re: [ntfs-3g-devel] Experimental support for Windows 10 "System Compressed" files

2015-11-27 Thread Eric Biggers
On Wed, Nov 25, 2015 at 10:32:06AM +0100, Jean-Pierre André wrote: > Agreed. It should be plugin.h, but where should this be > located ("src/plugin.h" ?) There are a few options I can think of: 1.) Make it include/ntfs-3g/plugin.h and install it with the library headers 2.) Make it src/plugin.h

Re: [ntfs-3g-devel] Experimental support for Windows 10 "System Compressed" files

2015-11-20 Thread Eric Biggers
age. Eric On Tue, Sep 22, 2015 at 10:54:10PM -0500, Eric Biggers wrote: > I've pushed changes to my repository that address a few things you brought > up: > > - compiler warnings addressed > - decompression memory allocated on heap rather than stack > - a couple optimizations f

[ntfs-3g-devel] [PATCH 2/2] Conditionally compile debugging code in ntfs_delete()

2016-06-21 Thread Eric Biggers
Although ntfs_log_trace() is defined to a no-op in non-DEBUG builds, ntfs_attr_name_get() is not. This function performs a string conversion and a memory allocation, so it is nice to have the call to it compiled out when not needed. Signed-off-by: Eric Biggers <ebigge...@gmail.com> --- l

[ntfs-3g-devel] [RFC PATCH] Always open $Secure when mounting NTFS volume

2016-07-26 Thread Eric Biggers
tfs_volume. Some memory leaks in error paths of ntfs_open_secure() are also fixed. Signed-off-by: Eric Biggers <ebigge...@gmail.com> --- include/ntfs-3g/security.h | 4 ++- libntfs-3g/security.c | 87 -- libntfs-3g/volume.c| 7

[ntfs-3g-devel] [PATCH] xattrs.c: remove unused variables

2016-07-26 Thread Eric Biggers
Signed-off-by: Eric Biggers <ebigge...@gmail.com> --- libntfs-3g/xattrs.c | 4 1 file changed, 4 deletions(-) diff --git a/libntfs-3g/xattrs.c b/libntfs-3g/xattrs.c index f17e4ca..2b7e709 100644 --- a/libntfs-3g/xattrs.c +++ b/libntfs-3g/xattrs.c @@ -81,10 +81,6 @@ struct LE_POS

[ntfs-3g-devel] [PATCH] ntfscmp: fix tautological comparison

2016-07-26 Thread Eric Biggers
Signed-off-by: Eric Biggers <ebigge...@gmail.com> --- ntfsprogs/ntfscmp.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ntfsprogs/ntfscmp.c b/ntfsprogs/ntfscmp.c index 555e401..cabc9c0 100644 --- a/ntfsprogs/ntfscmp.c +++ b/ntfsprogs/ntfscmp.c @@ -547,7 +547,7 @@ stati

[ntfs-3g-devel] [PATCH] Correct validation of multi sector transfer protected records

2016-07-27 Thread Eric Biggers
d move it into a helper function, as it was done identically in the read and write paths. Signed-off-by: Eric Biggers <ebigge...@gmail.com> --- libntfs-3g/mst.c | 45 +++-- 1 file changed, 27 insertions(+), 18 deletions(-) diff --git a/libntfs-3g/m

Re: [ntfs-3g-devel] [PATCH] Correct validation of multi sector transfer protected records

2016-07-27 Thread Eric Biggers
On Wed, Jul 27, 2016 at 12:20:24PM +0200, Jean-Pierre André wrote: > > Can you disambiguate the word "sector" here ? This is not > a physical sector, but an ntfs logical sector whose size > is NTFS_BLOCK_SIZE (512 bytes). This might not have been > known to the original developer, and it would be

[ntfs-3g-devel] [PATCH] Filename collation cleanups

2016-07-26 Thread Eric Biggers
Signed-off-by: Eric Biggers <ebigge...@gmail.com> --- include/ntfs-3g/layout.h | 33 - libntfs-3g/unistr.c | 28 +++- 2 files changed, 31 insertions(+), 30 deletions(-) diff --git a/include/ntfs-3g/layout.h b/include/ntfs-3g/layout.h

[ntfs-3g-devel] New repository for system compression plugin

2016-07-02 Thread Eric Biggers
Hello, I have made the NTFS-3G system compression plugin available in a new repository at https://github.com/ebiggers/ntfs-3g-system-compression. I also made a few small updates and updated the build system to use autotools. With libntfs-3g installed including headers, the plugin can be built

Re: [ntfs-3g-devel] [PATCH 2/2] ACE validation fixes

2016-09-25 Thread Eric Biggers
On Wed, Sep 21, 2016 at 12:01:23PM +0200, Jean-Pierre André wrote: > > I have never met an object ACE and they might be irrelevant > for a file system which only deals with files and directories. > > Is there a point in ntfs-3g accepting ACE types controlling > entities which are not emulated on

[ntfs-3g-devel] [PATCH 2/2] Remove unused argument from ntfs_make_symlink()

2016-09-21 Thread Eric Biggers
Now that the size of the reparse point attribute is no longer used by the FUSE drivers to populate st_size for symlinks and junctions, it no longer needs to be returned by ntfs_make_symlink(). Signed-off-by: Eric Biggers <ebigge...@gmail.com> --- include/ntfs-3g/reparse.h | 4 ++-- libn

[ntfs-3g-devel] [PATCH 1/2] ntfs-3g, lowntfs-3g: set correct st_size for symlinks

2016-09-21 Thread Eric Biggers
NTFS-3G used several different conventions for setting st_size of symlinks. Make it use the standard POSIX convention of setting st_size to the length of the link target without a terminating null. Signed-off-by: Eric Biggers <ebigge...@gmail.com> --- src/lowntfs-3g.

Re: [ntfs-3g-devel] [PATCH 2/2] Remove unused argument from ntfs_make_symlink()

2016-09-21 Thread Eric Biggers
On Wed, Sep 21, 2016 at 10:47:57AM +0200, Jean-Pierre André wrote: > Hi Eric, > > There has been a recent request for ntfs-3g to return > the st_size for symlinks as the size of the target > path (as described in the stat manual), so the target > is now useful > > Regards > Well, the

[ntfs-3g-devel] [PATCH] reparse.c: validate minimum size of mountpoint/symlink reparse points

2016-09-15 Thread Eric Biggers
valid_reparse_data() would read past the end of the reparse point buffer if it was passed a malformed reparse point that had the tag for a mountpoint or a symlink but had a data buffer smaller than expected. Fix this by validating the buffer size. Signed-off-by: Eric Biggers <ebigge...@gmail.

[ntfs-3g-devel] [PATCH] unistr.c: fix another buffer overrun in ntfs_utf16_to_utf8()

2016-09-15 Thread Eric Biggers
length. This was a longstanding bug. Fix it by leaving space for a terminating null. Signed-off-by: Eric Biggers <ebigge...@gmail.com> --- libntfs-3g/unistr.c | 14 ++ 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/libntfs-3g/unistr.c b/libntfs-3g/unistr.c index 1

[ntfs-3g-devel] [PATCH] unistr.c: make utf16_to_utf8_size() always honor @outs_len

2016-09-15 Thread Eric Biggers
utf16_to_utf8_size() was not guaranteed to fail with ENAMETOOLONG if the computed length was greater than @outs_len. This could cause a buffer overrun in ntfs_utf16_to_utf8(). This was a bug introduced by the patches to allow broken Unicode. Fix it. Signed-off-by: Eric Biggers <ebi

[ntfs-3g-devel] [PATCH] Eliminate NTFS_BUG()

2016-09-15 Thread Eric Biggers
NTFS_BUG() was broken because it relied on dereferencing a NULL pointer. This is undefined behavior, and gcc was compiling out the statement. Crashing in library code is also unfriendly in general. There were only two users. Make them just use regular error handling. Signed-off-by: Eric Biggers

[ntfs-3g-devel] [PATCH] unistr.c: remove unused function ntfs_file_value_upcase()

2016-09-15 Thread Eric Biggers
ntfs_file_value_upcase() is not called from anywhere in NTFS-3G, seems unlikely to be used by third-party programs, and can be replaced with calling ntfs_name_upcase() directly. So remove it. Signed-off-by: Eric Biggers <ebigge...@gmail.com> --- include/ntfs-3g/unistr.h | 3 --- libn

[ntfs-3g-devel] [PATCH 1/2] Add definitions of ACE types up to Windows 10

2016-09-21 Thread Eric Biggers
ACEs can be found in the public documentation on MSDN. Signed-off-by: Eric Biggers <ebigge...@gmail.com> --- include/ntfs-3g/layout.h | 90 +--- 1 file changed, 62 insertions(+), 28 deletions(-) diff --git a/include/ntfs-3g/layout.h b/include/n

[ntfs-3g-devel] [PATCH 2/2] ACE validation fixes

2016-09-21 Thread Eric Biggers
bject ACEs are rarely used.) - Only validate the SID for recognized ACE types. The placement or presence of the SID should not be assumed for future ACE types. Signed-off-by: Eric Biggers <ebigge...@gmail.com> --- libntfs-3g/acls.c | 114 +++---

Re: [ntfs-3g-devel] [PATCH] unistr.c: make utf16_to_utf8_size() always honor @outs_len

2016-10-29 Thread Eric Biggers
On Sat, Oct 29, 2016 at 12:07:17PM +0200, Jean-Pierre André wrote: > Eric Biggers wrote: > > On Sat, Oct 29, 2016 at 09:45:57AM +0200, Jean-Pierre André wrote: > > > > > > I am waiting for a green light from Tuxera for merging them > > > into the git. > &g

Re: [ntfs-3g-devel] [PATCH 2/2] ACE validation fixes

2016-10-29 Thread Eric Biggers
On Sat, Oct 29, 2016 at 11:21:37AM +0200, Jean-Pierre André wrote: > Hi again, > > Eric Biggers wrote: > > Hi Jean-Pierre, > > > > Sorry for the late response. > > No problem. I also did not do much about it. > > The intent of ntfs_valid_descr() was

[ntfs-3g-devel] [PATCH] ntfs-3g, lowntfs-3g: remove failed_secure variable

2016-10-29 Thread Eric Biggers
This is no longer used. Signed-off-by: Eric Biggers <ebigge...@gmail.com> --- src/lowntfs-3g.c | 3 --- src/ntfs-3g.c| 3 --- 2 files changed, 6 deletions(-) diff --git a/src/lowntfs-3g.c b/src/lowntfs-3g.c index 9d933d2..59fbcd0 100644 --- a/src/lowntfs-3g.c +++ b/src/lowntf

[ntfs-3g-devel] [PATCH] Allow setting DOS name when long name has trailing dot or space

2016-11-20 Thread Eric Biggers
. But continue to forbid creating such names on a filesystem FUSE-mounted with the windows_name option. Additionally, continue to forbid a trailing a dot or space in DOS names; this matches the Windows behavior. Signed-off-by: Eric Biggers <ebigge...@gmail.com> --- include/ntfs-3g/layout.

Re: [ntfs-3g-devel] [PATCH 2/2] ACE validation fixes

2016-10-28 Thread Eric Biggers
Hi Jean-Pierre, Sorry for the late response. On Mon, Sep 26, 2016 at 01:52:36PM +0200, Jean-Pierre André wrote: > > 1. "Object" ACEs are mentioned as only being used for Active Directory > > objects > > [source: Windows Internals 6th edition]. On Windows, trying to use > >

Re: [ntfs-3g-devel] [PATCH] unistr.c: make utf16_to_utf8_size() always honor @outs_len

2016-10-28 Thread Eric Biggers
nd a patch for that too. Thanks, Eric On Wed, Sep 14, 2016 at 11:39:07PM -0700, Eric Biggers wrote: > utf16_to_utf8_size() was not guaranteed to fail with ENAMETOOLONG if the > computed length was greater than @outs_len. This could cause a buffer > overrun in ntfs_utf16_to_utf8(). This w

Re: [ntfs-3g-devel] [PATCH] lowntfs-3g: correctly pass file info to reparse plugins

2016-10-29 Thread Eric Biggers
nment which defines uintptr_t different from unsigned > long ?" > > As doing something about it in "configure" would be > overkilling, I will probably take the proposal for next > version. > > Jean-Pierre > > Eric Biggers wrote: > > Signed-off-by: E

[ntfs-3g-devel] [PATCH] attrib.c: fix expanding $STANDARD_INFORMATION with almost-full MFT record

2016-12-30 Thread Eric Biggers
bug was found using the wlfuzz program from wimlib. Signed-off-by: Eric Biggers <ebigge...@gmail.com> --- libntfs-3g/attrib.c | 4 1 file changed, 4 insertions(+) diff --git a/libntfs-3g/attrib.c b/libntfs-3g/attrib.c index a5a6549a..1cc3ef64 100644 --- a/libntfs-3g/attrib.c +++ b/libn