Re: [OpenAFS] OpenAFS Windows build environment
On Tue, Feb 14, 2017 at 03:14:30PM +0200, Kostas Liakakis wrote: > Hello Ben, > > > So, where to now? Is there a way to actually test the resulting binaries > as they are? Try a newer WiX version and see where it bombs? Upload the > build result somewhere for you to check it? It may be easiest to test the resulting binaries "in-place", or perhaps on a separate scratch VM that can be wiped clean easily (as opposed to the build VM itself, which may require effort to replace, depending on the state of snapshotting available, etc.). E.g., on win7 you can boot into developer mode that does not require any signatures on the kernel modules at all, which suffices to test functionality of the build. Of course, openafs in general is lacking in automated test coverage, so the first step might just be manual testing. (Writing automated testing would be a great second step!) I have no interest in trying out the build results myself, so don't upload them just for my benefit. (Maybe there are other volunteers, though? They should feel free to speak up if there is interest.) I also have little interest in rewriting the installer with only email feedback on how things work. So, trying out the build with a newer WiX is probably only worth it if you (or someone you know) is interested in learning WiX and working through the needed changes. http://wixtoolset.org/documentation/manual/v3/ would probably be the relevant documentation landing point, alongside reading through some existing installer XML to see what is used in practice. (That is not necessarily the existing openafs installer XML, which as we have established is essentially unmaintained and unlikely to be a great example. I don't have a great sense for how many open-source projects are using WiX, but maybe one of the Kerberos distributions would be a useful example. MIT's installer would be around https://github.com/krb5/krb5/tree/master/src/windows/installer/wix ... maybe Jeffrey has a link handy to the Heimdal equivalent?) -Ben ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] OpenAFS Windows build environment
On Tue, Feb 14, 2017 at 11:51:19AM -0500, Jeffrey Altman wrote: > On 2/14/2017 7:37 AM, Kostas Liakakis wrote: > > > > Hi Jeffrey, > > > > On 2017-02-14 03:53, Jeffrey Altman wrote: > >> They are built with WiX 3.9 scripts. The > >> installation packaging in the OpenAFS tree can no longer be used. > > I see. So I'll switch to a later version as Ben suggested and see what > > happens. Hopefully we won't have to reinvent the wheel. > > The WiX 2.0 scripts cannot be used. Someone will have to write a new > installer. Well, perhaps there is something of a ship of Theseus when "new" is used...someone will certainly need to read up on WiX and understand what is going on, at least. It's a little strange coming from my background, but IIRC the documentation was fairly reasonable. -Ben ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] OpenAFS Windows build environment
On Tue, Feb 14, 2017 at 09:56:19AM -0500, Jeffrey Altman wrote: > On 2/14/2017 8:14 AM, Kostas Liakakis wrote: > > The IDN is required when building for pre-Vista because you have > to install the normaliz.dll library as part of the installer. I would posit that we should not attempt to support pre-Vista, given the difficulty and our available manpower. (Also, probably not Vista itself, either.) -Ben ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] OpenAFS Windows build environment
On 2/14/2017 12:15 PM, Kostas Liakakis wrote:
> Even in 1.8.0pre1 tarball the normalize.h inclusion in cm_nls.c is
> unconditional and in Platform SDK 6.0a a quick search comes back empty
> for NORM_FORM.
[C:\Program Files\Microsoft SDKs\Windows\v6.0\Include]grep _NORM_ *
WinNls.h:typedef enum _NORM_FORM {
[C:\Program Files\Microsoft SDKs\Windows\v6.0A\Include]grep _NORM_ *
WinNls.h:typedef enum _NORM_FORM {
[C:\Program Files\Microsoft SDKs\Windows\v7.0\Include]grep _NORM_ *
WinNls.h:typedef enum _NORM_FORM {
[C:\Program Files\Microsoft SDKs\Windows\v8.1\Include\um]grep _NORM_ *
WinNls.h:typedef enum _NORM_FORM {
etc...
> And maybe, since you probably
> have a solid and current build environment already setup, could you
> please share your tool and SDK versions so I can match them instead of
> trying to solve problems already dealt with?
If I thought there was benefit in maintaining an out of date tool chain
I would not have shutdown the prior builders.
Jeffrey Altman
<>
smime.p7s
Description: S/MIME Cryptographic Signature
Re: [OpenAFS] OpenAFS Windows build environment
On 2017-02-14 16:56, Jeffrey Altman wrote:
> On 2/14/2017 8:14 AM, Kostas Liakakis wrote:
>> Well, I gave up on it :)
>> Instead commented the #include in cm_nls.c and added a typedef for
>> _NORM_FORM enum copied out of the MS documentation.
> The NORM_FORM enum is defined in the Windows SDK. The normalization.h
> include is protected by a conditional
>
> #if (_WIN32_WINNT < 0x0600)
> /* This is part of the Microsoft Internationalized Domain Name
> Mitigation APIs. */
> # include
> #endif
>
> which matches the conditional in the SDK headers:
>
> #if (WINVER >= 0x0600)
> //
> // Normalization forms
> //
>
> typedef enum _NORM_FORM {
> ...
> } NORM_FORM;
What version of the OpenAFS source tree and Windows Platform SDK are you
quotting from?
Even in 1.8.0pre1 tarball the normalize.h inclusion in cm_nls.c is
unconditional and in Platform SDK 6.0a a quick search comes back empty
for NORM_FORM.
I started with a "known good" source tree and tried to replicate a known
good build environment as suggested in the guide I was pointed at as a
starting point. So I picked 1.7.32 and tried to match all the toolchain
versions as much as possible.
Should I have cloned the git tree instead? And maybe, since you probably
have a solid and current build environment already setup, could you
please share your tool and SDK versions so I can match them instead of
trying to solve problems already dealt with?
>> Amazingingly, this was enough for the build to proceed and end in
>> success.
> The cm_nls.c uses run-time loading of the functions so that the code can
> be compiled for versions of Windows that do not ship with the
> NormalizeString() and IsNormalizedString() functions.
>
> The IDN is required when building for pre-Vista because you have
> to install the normaliz.dll library as part of the installer.
>
>> Well, at least my openafs/dest directory is now populated and I
>> could at least run "pts help" and "fs help".
> Neither pts nor fs use the normalization routines directly.
Oh, I see.
But anyway, what I wanted to say is that in the end I did manage to get
executables that execute... to some extent at least.
-K.
smime.p7s
Description: S/MIME Cryptographic Signature
Re: [OpenAFS] OpenAFS Windows build environment
On 2/14/2017 7:37 AM, Kostas Liakakis wrote: > > Hi Jeffrey, > > Thanks for taking the time to answer. Please read below inline. > > > On 2017-02-14 03:53, Jeffrey Altman wrote: >> They are built with WiX 3.9 scripts. The >> installation packaging in the OpenAFS tree can no longer be used. > I see. So I'll switch to a later version as Ben suggested and see what > happens. Hopefully we won't have to reinvent the wheel. The WiX 2.0 scripts cannot be used. Someone will have to write a new installer. >> As a reminder, the 1.7.3301 installers that AuriStor, Inc. distributes >> >> https://www.auristor.com/openafs/client-installer/ >> >> can be installed on Windows 10 and Windows Server 2016 because they are >> grand-fathered. If the same sources were built today they would not >> produce a working file system. > > This statement is a bit disturbing. But should not be surprising since I gave the community plenty of warning: https://lists.openafs.org/pipermail/openafs-info/2015-March/041324.html https://lists.openafs.org/pipermail/openafs-info/2015-April/041325.html https://lists.openafs.org/pipermail/openafs-info/2015-April/041328.html https://lists.openafs.org/pipermail/openafs-info/2015-April/041330.html https://lists.openafs.org/pipermail/openafs-info/2015-April/041332.html https://lists.openafs.org/pipermail/openafs-info/2015-June/041392.html https://lists.openafs.org/pipermail/openafs-info/2015-July/041449.html http://workshop.openafs.org/afsbpw15/talks/thursday/AFS-on-Windows-AFSBPW15.pdf http://workshop.openafs.org/afsbpw15/talks/friday/Securing_The_OS.pdf > Do you mean that given the state of things, even if Windows buildbots > start springing back to life and binaries for current OpenAFS versions > become available, no later version than the already built and signed > 1.7.3301 can be installed on Win10 and later? Will Win7 be ok at least? I've answered these questions in the e-mails and presentations listed above. > Do you consider the whole effort to revive the Windows port is in vain? In April 2015 I provided estimates of what I believe the on-going costs are for obtaining and maintaining a Microsoft signature for a file system. (Second link above.) Last week I attended IFS PlugFest 29. Microsoft is serious about improving the reliability of Windows and its resistance to root kits, ransomeware, and other forms of malware. Each year the requirements that driver vendors must satisfy become more demanding. * Mandatory to implement functionality * Mandatory to use build chains * Mandatory to submit testing reports for each OS variant on which the driver might be installed * Mandatory use of EV code signing certificates The challenge for the OpenAFS community is that the requirements cannot be satisfied simply by compiling the Windows client with the latest tool chains. The mandatory to implement functionality requires support from AFS (VL, FILE, RX, ...) that simply does not exist today. With the release of Windows 10 Creators Update the bar will be raised once again. > Or is there something the the Foundation is planning to do in order to > move forward again on the Windows platform? I am not a member of the Foundation Board. Their minutes can be reviewed at http://www.openafsfoundation.org/about/minutes/ Perhaps a board member could comment on their plans. Jeffrey Altman <> smime.p7s Description: S/MIME Cryptographic Signature
Re: [OpenAFS] OpenAFS Windows build environment
On 2/14/2017 8:14 AM, Kostas Liakakis wrote:
> Well, I gave up on it :)
> Instead commented the #include in cm_nls.c and added a typedef for
> _NORM_FORM enum copied out of the MS documentation.
The NORM_FORM enum is defined in the Windows SDK. The normalization.h
include is protected by a conditional
#if (_WIN32_WINNT < 0x0600)
/* This is part of the Microsoft Internationalized Domain Name
Mitigation APIs. */
# include
#endif
which matches the conditional in the SDK headers:
#if (WINVER >= 0x0600)
//
// Normalization forms
//
typedef enum _NORM_FORM {
...
} NORM_FORM;
> Amazingingly, this was enough for the build to proceed and end in
> success.
The cm_nls.c uses run-time loading of the functions so that the code can
be compiled for versions of Windows that do not ship with the
NormalizeString() and IsNormalizedString() functions.
The IDN is required when building for pre-Vista because you have
to install the normaliz.dll library as part of the installer.
> Well, at least my openafs/dest directory is now populated and I
> could at least run "pts help" and "fs help".
Neither pts nor fs use the normalization routines directly.
Jeffrey Altman
<>
smime.p7s
Description: S/MIME Cryptographic Signature
Re: [OpenAFS] OpenAFS Windows build environment
Hello Ben, On 2017-02-14 07:45, Benjamin Kaduk wrote: > (FWIW, there is > currently a windows builder that was set up by Alejandro Sedeno > on a VM donated by MIT: > https://buildbot.openafs.org/buildslaves/win7-amd64 . But there is It would be nice to know what software versions Alejandro used for his slave. Taking a quick look in his build log I can imagine he has VS2005 and Platform SDK 7.0 installed. In contrast, I went with VS2008 and Platform SDK 6.0a, trying to be as close to the Secure Endpoints mentioned setup as possible and work from there. > The binaries produced on the buildbots are not distributed, yes. Good :) > It seems that at least sometimes, the historical usage has been to > have a different ntbuild.bat for each of the four combinations of > 32/64-bit and debug/release builds (sometimes called free/checked). > But it probably doesn't matter which one you start off with. At least for the 32/64 bit variation it seems that a separate ntbuild.bat should be used, or the current one be coded better. For the debug/release one there seems to be a command line parameter supported: free/checked > > That one may be a bit harder to find. (I don't even know what the > redistributable status of such a thing would be even if someone did > have a copy sitting around...) Well, I gave up on it :) Instead commented the #include in cm_nls.c and added a typedef for _NORM_FORM enum copied out of the MS documentation. Amazingingly, this was enough for the build to proceed and end in success. Well, at least my openafs/dest directory is now populated and I could at least run "pts help" and "fs help". Haven't bother installing the tools for building the doc tree. I don't think Alejandro is either... So, where to now? Is there a way to actually test the resulting binaries as they are? Try a newer WiX version and see where it bombs? Upload the build result somewhere for you to check it? Cheers, -K. smime.p7s Description: S/MIME Cryptographic Signature
Re: [OpenAFS] OpenAFS Windows build environment
Hi Jeffrey, Thanks for taking the time to answer. Please read below inline. On 2017-02-14 03:53, Jeffrey Altman wrote: > They are built with WiX 3.9 scripts. The > installation packaging in the OpenAFS tree can no longer be used. I see. So I'll switch to a later version as Ben suggested and see what happens. Hopefully we won't have to reinvent the wheel. > As a reminder, the 1.7.3301 installers that AuriStor, Inc. distributes > > https://www.auristor.com/openafs/client-installer/ > > can be installed on Windows 10 and Windows Server 2016 because they are > grand-fathered. If the same sources were built today they would not > produce a working file system. This statement is a bit distrurbing. Do you mean that given the state of things, even if Windows buildbots start springing back to life and binaries for current OpenAFS versions become availble, no later version than the already built and signed 1.7.3301 can be installed on Win10 and later? Will Win7 be ok at least? Do you consider the whole effort to revive the Windows port is in vain? Or is there something the the Foundation is planning to do in order to move forward again on the Windows platform? Thanks, -K. smime.p7s Description: S/MIME Cryptographic Signature
Re: [OpenAFS] OpenAFS Windows build environment
On Mon, Feb 13, 2017 at 10:58:28PM +0200, Kostas Liakakis wrote: > > I 've decided (again) to take the plunge and attempt to create a build > bot slave for OpenAFS on Windows platform. We use OpenAFS on our Windows > PC labs and would like to continue to do so. Thank you for stepping forward to help! It is good to have more people who are familiar with the windows setup. (FWIW, there is currently a windows builder that was set up by Alejandro Sedeno on a VM donated by MIT: https://buildbot.openafs.org/buildslaves/win7-amd64 . But there is probably still value in continuing your attempts -- for one, we are limited to only two builds at a time on that machine, and there are potentially four that are worth doing -- only two are in the set of builds that are triggered for each submitted change.) > I have been following the guide found here: > http://git.openafs.org/?p=openafs.git;a=blob;f=README-WINDOWS;h=4f9af252f51677dd49e32c4af4411bd96eed53c9;hb=HEAD > > The guide itself admits it might be a bit outdated at it really is... Yes ... and it is not entirely sure even how outdated it is (I say, having made the most recent overhaul to it.) > First things first, it is stated that only VS 2005 or 2008 is supported. > The earliest VS I can have a valid license for is 2012. And that license > would be through our University's Microsoft Imagine (ex Dreamspark) > subscription. As I believe the binaries produced on a buildbot aren't > the actual ones distributed I suppose a license can be used for this > purpose. If not, I hope somebody can give some more insight on this. The binaries produced on the buildbots are not distributed, yes. > But since, VS 2012 is a big step forward, for the time being I settled > for the 90day trial of VS2008 I found available for download from MS. > According to the guide, Secure Endpoints Inc were building distributed > binaries using VS2008. In my attempts (several years ago, now), I do not think I actually succeeded in getting a build with VS2008 to work (only VS2005). But that was a long time ago, and my notes are incomplete, so it's hard to be sure. > I decided to target an AMD64 build on Windows 7 x64 for starters. Soon I > found out that NTbuild.bat alters PATH and the equivalent of Windows > LIBPATH. Specifically, these lines > > set AFSDEV_LIB=%MSSDKDIR%\lib;%MSVCDIR%\lib; > set AFSDEV_BIN=%MSSDKDIR%\bin;%MSVCDIR%\bin;%PERL%\bin;%CYGWINDIR%\bin;%WIX% > > cause the 32bit flavors of VS compiler and libraries to be used, even if > an AMD64 build is requested according to the guide linked above. > > The correct settings should be: > > set AFSDEV_LIB=%MSSDKDIR%\lib\x64;%MSVCDIR%\lib\amd64; > set > AFSDEV_BIN=%MSSDKDIR%\bin;%MSVCDIR%\bin\amd64;%PERL%\bin;%CYGWINDIR%\bin;%WIX% It seems that at least sometimes, the historical usage has been to have a different ntbuild.bat for each of the four combinations of 32/64-bit and debug/release builds (sometimes called free/checked). But it probably doesn't matter which one you start off with. > With that out of the way, the build continues up until Microsoft IDN > APIs headers are required. For the life of me I couldn't locate any > place where this is available. All Google links I found are a dead-end. > Is there anybody who has a source for IDN APIs 1.1 ? That one may be a bit harder to find. (I don't even know what the redistributable status of such a thing would be even if someone did have a copy sitting around...) > And then the next problem is that apparently WiX 2.0.5805.0 will not > recognize VS2008 Professional and refuse to install. Some digging around > revealed forum posts stating that WiX 2.0 is too old to support VS2008. > But then the guide states that all newer versions WiX are a no-no... > which leads to a chicken and egg situation... IIRC the WiX issue is what drove me off of VS2008 in my experimentation so long ago. But, ultimately, our dependence on outdated/unsupported/no-longer-available tools will become unsustainable, and we will be forced to upgrade. It would be better to pay the burden of that upgrade now, when it can be done at our leisure, than have to do it in a hurry. So, maybe it is the time to ignore the documented requirements and see what breaks when using a newer WiX. (Obviously, start off with the oldest WiX that is compatible with the old VS2008 -- don't try to jump straight to the latest!) And, of course, WiX is only needed for building installers; just compiling the code can be done without any WiX at all. > Does anybody have any knowledge of what version of WiX did Secure > Endpoints used in their builds, provided the information that they had > been using VS2008 is correct? I think Jeffrey has already answered that one. If you are not doing so already, I recommend keeping a "lab notebook" that records what tools you installed, what commands you used to (attempt to) build, etc., which could eventually be used to make the above README actually resemble reality. Thanks again,
Re: [OpenAFS] OpenAFS Windows build environment
On 2/13/2017 3:58 PM, Kostas Liakakis wrote: > Does anybody have any knowledge of what version of WiX did Secure > Endpoints used in their builds, provided the information that they had > been using VS2008 is correct? Kostas, The OpenAFS 1.7.3301 installers distributed by AuriStor, Inc. are built from OpenAFS sources but they do not use the OpenAFS WiX 2.0 scripts for the installation packages. They are built with WiX 3.9 scripts. The installation packaging in the OpenAFS tree can no longer be used. As a reminder, the 1.7.3301 installers that AuriStor, Inc. distributes https://www.auristor.com/openafs/client-installer/ can be installed on Windows 10 and Windows Server 2016 because they are grand-fathered. If the same sources were built today they would not produce a working file system. Jeffrey Altman <> smime.p7s Description: S/MIME Cryptographic Signature
[OpenAFS] OpenAFS Windows build environment
Hello, I 've decided (again) to take the plunge and attempt to create a build bot slave for OpenAFS on Windows platform. We use OpenAFS on our Windows PC labs and would like to continue to do so. I have been following the guide found here: http://git.openafs.org/?p=openafs.git;a=blob;f=README-WINDOWS;h=4f9af252f51677dd49e32c4af4411bd96eed53c9;hb=HEAD The guide itself admits it might be a bit outdated at it really is... First things first, it is stated that only VS 2005 or 2008 is supported. The earliest VS I can have a valid license for is 2012. And that license would be through our University's Microsoft Imagine (ex Dreamspark) subscription. As I believe the binaries produced on a buildbot aren't the actual ones distributed I suppose a license can be used for this purpose. If not, I hope somebody can give some more insight on this. But since, VS 2012 is a big step forward, for the time being I settled for the 90day trial of VS2008 I found available for download from MS. According to the guide, Secure Endpoints Inc were building distributed binaries using VS2008. I decided to target an AMD64 build on Windows 7 x64 for starters. Soon I found out that NTbuild.bat alters PATH and the equivalent of Windows LIBPATH. Specifically, these lines set AFSDEV_LIB=%MSSDKDIR%\lib;%MSVCDIR%\lib; set AFSDEV_BIN=%MSSDKDIR%\bin;%MSVCDIR%\bin;%PERL%\bin;%CYGWINDIR%\bin;%WIX% cause the 32bit flavors of VS compiler and libraries to be used, even if an AMD64 build is requested according to the guide linked above. The correct settings should be: set AFSDEV_LIB=%MSSDKDIR%\lib\x64;%MSVCDIR%\lib\amd64; set AFSDEV_BIN=%MSSDKDIR%\bin;%MSVCDIR%\bin\amd64;%PERL%\bin;%CYGWINDIR%\bin;%WIX% With that out of the way, the build continues up until Microsoft IDN APIs headers are required. For the life of me I couldn't locate any place where this is available. All Google links I found are a dead-end. Is there anybody who has a source for IDN APIs 1.1 ? And then the next problem is that apparently WiX 2.0.5805.0 will not recognize VS2008 Professional and refuse to install. Some digging around revealed forum posts stating that WiX 2.0 is too old to support VS2008. But then the guide states that all newer versions WiX are a no-no... which leads to a chicken and egg situation... Does anybody have any knowledge of what version of WiX did Secure Endpoints used in their builds, provided the information that they had been using VS2008 is correct? Thanks for reading so far down! -K. smime.p7s Description: S/MIME Cryptographic Signature
