Hi,
I have just completed a fork of ntfs-3g that adds support for building
the project as UEFI driver.
As such, I would very much like to see this feature included in
mainline, especially I have taken great care to break down the commits
into a patch series that should facilitate its review for integration.
You can find the fork at: https://github.com/pbatard/ntfs-3g
You can also find the ~65 commits that are needed to add UEFI driver
support starting around
https://github.com/pbatard/ntfs-3g/commits/uefi-driver?after=0e160412aa15b51f700c6d7f1175abba39233d3c+34&branch=uefi-driver.
With these patches applied, one can use the ntfs-3g source to build a
UEFI NTFS driver, for all the mainline UEFI architectures (X64, IA32,
AARCH64, ARM) on either Linux or Windows, through a gcc/EDK2 toolchain
(Linux) or a VS2019/EDK2 or VS2019/gnu-efi toolchain (Windows).
You can also find a very concrete example of how one can produce such a
build on Ubuntu through the AppVeyor build log at
https://ci.appveyor.com/project/pbatard/ntfs-3g?fullLog=true, which
details how we produce our current binary releases (that are also
currently found at https://github.com/pbatard/ntfs-3g/releases).
Furthermore, regardless of UEFI interest, I believe that our fork does
provide some features that mainline might want to consider for
integration, such as enabling one-shot mode for readdir (so that one can
halt the call after one entry has been processed and actually resume the
listing from the returned position) or static asserts in layout.h (which
can be an invaluable help when adding support for a new platform or new
toolchains, as it validates that the structure mapping does match the
NTFS requirements).
A detailed breakdown of the patch series is as follows:
* Patches 0001-0009 are mostly about fixing header issues, that may not
have been apparent when creating executables for POSIX OSes, but that
are a real problem when doing so for other OSes or low level UEFI.
* Patch 0010 fixes configure's --enable-debug option, which currently
does not translate to anything useful, in order to pre-emptively address
a conflict with the EDK2's DEBUG() macro.
* Patch 0011 adds a new option to force UTF-8 usage everywhere, required
for UEFI, as there is no such thing as POSIX-like locale support there.
* Patch 0012 adds NTFS volume serial number retrieval, which, while easy
to populate, was strangely missing from ntfs-3g. This is another
requirement for UEFI, as we need to be able to detect volume
removal/change, where the ability to check a volume serial becomes crucial.
* Patches 0013, as well as 0027-0033 deal with improving/fixing
compatibility of the codebase for MSVC compilers, as it is
understandably currently skewed towards to the use of gcc-like
compilers, and some of the gcc-isms don't sit too well with MSVC.
* Patches 0014-0017 start to handle UEFI specifics, with patch 0015
actually adding an initial dummy UEFI driver, that of course doesn't do
much of anything except loading/unloading, but can be built with the
EDK2. On that subject, it needs to be pointed out that the EDK2 has a
fairly "rigid" way of producing executables (you can't use
configure/make, but have to provide the .inf/.dsc/.dec/.uni files that
the toolchain expects) with some restrictions with regards to the
location of some of the files This is the reason why the uefi-driver.dec
and uefi-driver.dsc files must reside at the top level of the
repository, and cannot be "hidden" in the uefi-driver/ folder.
* Patch 0018 and 0020 fix conflicts between some elements that are
really handled as the same thing by EDK2 and ntfs-3g, but aren't seem
the same, on account of compiler pedantry.
* Patches 0019 and 0021-0026 and 0034-0037 deal mostly with adding the
required UEFI support functions to ntfs-3g
* Patches 0038-0047 relate to fleshing out the UEFI driver (there are
quite a few API layers to go through), until it can actually start to
call into libntfs-3g.
* Patch 0048 adds the UEFI I/O support layer to libntfs-3g,
* Patches 0049-0051, with 0050 being the one that adds generic static
assert support to ntfs-3g, have to do with ensuring that MVSC does pack
the layout structures properly.
* Patches 0052-0055 simplify readdir and add a "one-shot" mode, where
returning a positive value in the filldir callback suspends readdir
operation (which returns with a pos value pointing to the next entry),
so that the listing can be continued in a subsequent call. The reason
why we need a "one-shot" feature is that the UEFI expects repeated
formal API calls (not callbacks) for each directory entry, which, if we
didn't modify readdir, would force us to either issue a complete
directory listing for each entry we want to process or cache entries,
which isn't too great in terms of performance.
* Patches series 0056-0058 and 0059-0062 finalize the UEFI driver for
read-only and read/write operations respectively.
After that comes a series of 6 patches that I have tagged "(OPTIONAL)",
though I really would like to see at least 2 of those (VS2019 project
support and CompilersIntrinsicsLib) to be considered for formal
inclusion in ntfs-3g mainline, as these can be rather crucial for people
who are interested in building the UEFI driver.
* Patch 0063 adds an implementation of debug_double_inode(), which I
heavily relied on during testing of the driver, as ntfs-3g is rather
adamant that an inode should not be opened twice, whereas the UEFI shell
default behaviour makes no qualms about reopening the same file many
times over. On that subject, I should point out that a lot of the
complexity of the UEFI driver has to do with trying to avoid double
inode opening, with NtfsMoveFile() in uefi_bridge.c being quite the
challenge to get right...
* Patch 0064 adds a non GPLv2 (BSD-2-Clause-Patent) CompilerIntrinsics
library, without which it is currently not possible to build the X64 or
IA32 versions of the driver when using EDK2/MSVC. This library basically
deals with providing basic compilers intrinsics (low level div/mul +
some memory calls that the compiler insists on handling as intrinsics)
that cannot easily be defined in the ntfs-3g code. And whereas each of
these calls is rather short (though some contain assembly), it didn't
really make sense for us to rewrite a GPLv2 version when we could just
pick an existing BSD-2 implementation (which, per [1], is compatible
with GPLv2), though of course this means that we need to convey that the
this library is under a different license (which the patch does).
Despite the additional license, considering that not having this patch
makes EDK2 compilation on Windows impossible for X64 and IA32, I would
very much like to see it integrated.
* Patch 0065 adds gnu-efi as a submodule. gnu-efi [2] is a much leaner
alternative to EDK2, that can make the process of building UEFI
executables a lot less daunting for people who don't already have
experience with tianocore/EDK2. Especially, since it integrates well
with Visual Studio, it can lead to a complete turnkey solution, where
Windows users just have to open the project, after having cloned ntfs-3g
along with its submodules, to be able to both build and test the driver
by pressing VS2019's "Debug" button.
* Patch 0066 adds the Visual Studio 2019 project files. This is the
second "optional" patch that I would really like to see considered for
inclusion, as, perhaps to the future despair of current ntfs-3g project
maintainers, I suspect that quite a few folks will be interested in
being able to build the UEFI driver through Visual Studio, and I
certainly would hate to have invested as much time as I did, ensuring
that ntfs-3g and Visual Studio can play nice together, to have this
patch left aside.
* Patches 0067 and 0068 relate directly to the GitHub fork, and the
provision of pre-built binaries thus, unless ntfs-3g is actively
considering moving its whole project from SourceForge to GitHub, these
can be safely left out.
If you are interested in integrating these changes, I can send 0001-0066
as a formal patch series to this mailing list. Or I can sort something
in the SourceForge project, if you prefer.
But again, because a read/write UEFI NTFS driver is something that a lot
of people have been asking for (which I can vouch for, as I been
providing an alternate *read-only* version of such a driver for some
time, based on GRUB 2.0, at https://efi.akeo.ie) and also because
EDK2/UEFI doesn't seem to be moving towards providing an implementation,
despite counting multiple Microsoft developers as formal contributors, I
do think that it makes a lot of sense for ntfs-3g to want to officially
support UEFI as another platform. As such, I am really hopeful that the
ntfs-3g project maintainers will be interested in integrating these
changes into mainline.
Of course, I will be happy to answer any question you might have related
to this proposal.
Regards,
/Pete
[1] https://en.wikipedia.org/wiki/BSD_licenses
[2] https://sourceforge.net/projects/gnu-efi/
_______________________________________________
ntfs-3g-devel mailing list
ntfs-3g-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ntfs-3g-devel
