Re: [PATCH] sha1dc: update from upstream

[Michael Felt (aixtools)]

Sent from my iPhone

> On 2 Aug 2018, at 22:50, Ævar Arnfjörð Bjarmason  wrote:
> 
> Update sha1dc from the latest version by the upstream
> maintainer[1]. See 2db87328ef ("Merge branch 'ab/sha1dc'", 2017-07-10)
> for the last update.
> 
> This fixes an issue where AIX was wrongly detected as a Little-endian
> instead of a Big-endian system. See [2][3][4].
> 
> 1. 
> https://github.com/cr-marcstevens/sha1collisiondetection/commit/232357eb2ea0397388254a4b188333a227bf5b10
> 2. https://github.com/cr-marcstevens/sha1collisiondetection/pull/45
> 3. https://github.com/cr-marcstevens/sha1collisiondetection/pull/42
> 4. 
> https://public-inbox.org/git/20180729200623.gf945...@genre.crustytoothpaste.net/
> 
> Signed-off-by: Ævar Arnfjörð Bjarmason 
> ---
> 
>> On Wed, Aug 01 2018, Ævar Arnfjörð Bjarmason wrote:
>> https://github.com/cr-marcstevens/sha1collisiondetection/pull/45
>> [...]
>> It should work, but as noted in the MR please test it so we can make
>> sure, and then (if you have a GitHub account) comment on the MR saying
>> it works for you.
> 
> This got merged upstream, and as noted in that upstream PR I've
> personally tested this on AIX under both GCC and IBM's xlc on the GCC
> compile farm, it works.
> 
Thanks. I already have git 2.18 in use with the manual patch. 
> sha1collisiondetection |  2 +-
> sha1dc/sha1.c  | 12 +++-
> 2 files changed, 12 insertions(+), 2 deletions(-)
> 
> diff --git a/sha1collisiondetection b/sha1collisiondetection
> index 19d97bf5af..232357eb2e 16
> --- a/sha1collisiondetection
> +++ b/sha1collisiondetection
> @@ -1 +1 @@
> -Subproject commit 19d97bf5af05312267c2e874ee6bcf584d9e9681
> +Subproject commit 232357eb2ea0397388254a4b188333a227bf5b10
> diff --git a/sha1dc/sha1.c b/sha1dc/sha1.c
> index 25eded1399..df0630bc6d 100644
> --- a/sha1dc/sha1.c
> +++ b/sha1dc/sha1.c
> @@ -93,13 +93,23 @@
> #define SHA1DC_BIGENDIAN
> 
> /* Not under GCC-alike or glibc or *BSD or newlib or  */
> +#elif (defined(_AIX))
> +
> +/*
> + * Defines Big Endian on a whitelist of OSs that are known to be Big
> + * Endian-only. See
> + * 
> https://public-inbox.org/git/93056823-2740-d072-1ebd-46b440b33...@felt.demon.nl/
> + */
> +#define SHA1DC_BIGENDIAN
> +
> +/* Not under GCC-alike or glibc or *BSD or newlib or  
> or  */
> #elif defined(SHA1DC_ON_INTEL_LIKE_PROCESSOR)
> /*
>  * As a last resort before we do anything else we're not 100% sure
>  * about below, we blacklist specific processors here. We could add
>  * more, see e.g. https://wiki.debian.org/ArchitectureSpecificsMemo
>  */
> -#else /* Not under GCC-alike or glibc or *BSD or newlib or  whitelist>  or  */
> +#else /* Not under GCC-alike or glibc or *BSD or newlib or  whitelist> or  or  */
> 
> /* We do nothing more here for now */
> /*#error "Uncomment this to see if you fall through all the detection"*/
> -- 
> 2.18.0.345.g5c9ce644c3
> 

Re: Is detecting endianness at compile-time unworkable?

[Michael Felt]

I hope a I have a "leap forward"


On 7/30/2018 11:39 AM, Ævar Arnfjörð Bjarmason wrote:
> Perhaps it's worth taking a step back here and thinking about whether
> this whole thing is unworkable. It was hard enough to get this to work
> on the combination of Linux, *BSD and Solaris, but I suspect we'll run
> into increasingly obscure platforms where this is hard or impossible
> (AIX, HP/UX etc.)
While I still cannot say for HP/UX it does seem there is a potential
solution based on the status for _LITTLE_ENDIAN and _BIG_ENDIAN. At
least, gcc on POWER and xlc on POWER provides one or the other - and my
hope is that gcc on other platforms also provides them.

For "other" compilers that do not provide them - a modification to
CFLAGS to define one or the other should make "make" work.

Details (note - I am not a programmer, so by definition at least one of
my "macros" will be wrong :)

AIX and xlc
root@x066:[/]xlc   -qshowmacros -E /dev/null | grep -i endi
1506-297 (S) Unable to open input file null. A file or directory in the
path name does not exist..
#define __HHW_BIG_ENDIAN__ 1
#define __BIG_ENDIAN__ 1
#define __THW_BIG_ENDIAN__ 1
#define _BIG_ENDIAN 1

On SLES12 (le) and xlc
suse12test:~/images/littleEndian/sles # xlc -qshowmacros -dM -E x.c |
grep -i endi
#define _LITTLE_ENDIAN 1
#define __BYTE_ORDER__ __ORDER_LITTLE_ENDIAN__
#define __FLOAT_WORD_ORDER__ __ORDER_LITTLE_ENDIAN__
#define __LITTLE_ENDIAN__ 1
#define __ORDER_BIG_ENDIAN__ 4321
#define __ORDER_LITTLE_ENDIAN__ 1234
#define __ORDER_PDP_ENDIAN__ 3412
#define __VEC_ELEMENT_REG_ORDER__ __ORDER_LITTLE_ENDIAN__


Based on what I can see on gcc on POWER and xlc on POWER I think an
approach (simplified) can be:

#if undefined(_BIG_ENDIAN) && undef(_LITTLE_ENDIAN)
#error "one of _BIG_ENDIAN or _LITTLE_ENDIAN must be defined. Try adding
the correct value to CFLAGS"
#else defined(_BIG_ENDIAN) && defined(_LITTLE_ENDIAN)
#error "Only one of _BIG_ENDIAN and _LITTLE_ENDIAN may be defined, not both"
#endif

And then logic based on the value set.
This should also make cross-compile possible by unsetting an incorrect
default and setting the correct value.

p.s. Is there a setting I need to set somewhere so I receive a copy of
the email sent after it is received by the list. I could send myself a
copy, but I much prefer it comes from the maillist - as verification it
was received.

Re: Is detecting endianness at compile-time unworkable?

[Michael Felt]

On 7/30/2018 11:39 AM, Ævar Arnfjörð Bjarmason wrote:
> The reason we're in this hole is because we use this
> sha1collisiondetection library to do SHA-1, and the reason we have
> issues with it specifically (not OpenSSL et al) is because its only
> method of detecting endianness is at compile time.
When using gcc (no xlc available for Linux on Power)

POWER6 (Big Endian by definition)
root@x068:[/data/httpd/gcc]gcc -dM -E - < /dev/null | grep -i end
#define __ORDER_LITTLE_ENDIAN__ 1234
#define __BIG_ENDIAN__ 1
#define __FLOAT_WORD_ORDER__ __ORDER_BIG_ENDIAN__
#define __ORDER_PDP_ENDIAN__ 3412
#define _BIG_ENDIAN 1
#define __ORDER_BIG_ENDIAN__ 4321
#define __BYTE_ORDER__ __ORDER_BIG_ENDIAN__

SLES12 on POWER8
suse12test:~ # gcc -dM -E - < /dev/null | grep -i end
#define __ORDER_LITTLE_ENDIAN__ 1234
#define _LITTLE_ENDIAN 1
#define __FLOAT_WORD_ORDER__ __ORDER_LITTLE_ENDIAN__
#define __ORDER_PDP_ENDIAN__ 3412
#define __LITTLE_ENDIAN__ 1
#define __ORDER_BIG_ENDIAN__ 4321
#define __BYTE_ORDER__ __ORDER_LITTLE_ENDIAN__

*So, for compile time tests, when gcc is the compiler it seems the
following defines are available**
**__BIG_ENDIAN__, _BIG_ENDIAN,  __LITTLE__ENDIAN__, _LITTLE_ENDIAN**
**or something based on the value of __BYTE_ORDER__*

I'll see if I can find something similar for xlc, but will only be able
to test xlc on AIX.

>
> This didn't use to be the case, it was changed in this commit:
> https://github.com/cr-marcstevens/sha1collisiondetection/commit/d597672
>
> Dan Shumow: Since the commit message doesn't say why, can you elaborate
> a bit on why this was done, i.e. is determining this at runtime harmful
> for performance? If not, perhaps it would be best to bring this back, at
> least as an option.

Re: Is detecting endianness at compile-time unworkable?

[Michael Felt]

A small step back...


On 7/30/2018 11:39 AM, Ævar Arnfjörð Bjarmason wrote:

On Sun, Jul 29 2018, Michael wrote:


On 29/07/2018 22:06, brian m. carlson wrote:

On Sun, Jul 29, 2018 at 09:48:43PM +0200, Michael wrote:

On 29/07/2018 21:27, brian m. carlson wrote:

Well, that explains it.  I would recommend submitting a patch to
https://github.com/cr-marcstevens/sha1collisiondetection, and the we can
pull in the updated submodule with that fix.

Not sure I am smart enough to do that. I'll have to download, build, and see
what it says.

The issue is that somewhere in lib/sha1.c, you need to cause
SHA1DC_BIGENDIAN to be set.  That means you need to figure out what
compiler macro might indicate that.

I remember - roughly - a few decades back - having an assignment to
write code to determine endianness. PDP and VAC were different iirc,
and many other micro-processors besides the 8088/8086/z85/68k/etc..

If you are looking for a compiler macro as a way to determine this -
maybe you have one for gcc, but not for xlc. I do not know it - currently :)

I'm not familiar with AIX, but from searching around I found this
porting manual from IBM:
http://www.redbooks.ibm.com/redbooks/pdfs/sg246034.pdf
This is from July 2001 - when AIX 5L, for Linux affinity, was new. AIX 
was (nearly) the #1 posix system, and linux was a minor player in the 
data center (in or out (now as IAAS)). IMHO, the recommendations made in 
2001 are probably no longer applicable (64-bit was fairly new, e.g., 
rather than common).


There they suggest either defining your own macros, or testing the
memory layout at runtime (see section "2.2.2.3 Technique 3: Testing
memory layout" and surrounding sections).

Perhaps it's worth taking a step back here and thinking about whether
this whole thing is unworkable. It was hard enough to get this to work
on the combination of Linux, *BSD and Solaris, but I suspect we'll run
into increasingly obscure platforms where this is hard or impossible
(AIX, HP/UX etc.)

The reason we're in this hole is because we use this
sha1collisiondetection library to do SHA-1, and the reason we have
issues with it specifically (not OpenSSL et al) is because its only
method of detecting endianness is at compile time.
Cannot speak for the "others", but as I have mentioned before - as AIX 
in only on POWER it is also only Big Endian - so a compiletime #if 
testing for _AIX will work fine

This didn't use to be the case, it was changed in this commit:
https://github.com/cr-marcstevens/sha1collisiondetection/commit/d597672

Dan Shumow: Since the commit message doesn't say why, can you elaborate
a bit on why this was done, i.e. is determining this at runtime harmful
for performance? If not, perhaps it would be best to bring this back, at
least as an option.

And, as an aside, the reason we can't easily make it better ourselves is
because the build process for git.git doesn't have a facility to run
code to detect this type of stuff (the configure script is always
optional). So we can't just run this test ourselves.
On AIX - I am required to run configure, and frankly, I am amazed that 
not everyone is running it. Among other things I an modifying the prefix 
(to /opt) and many of the others to different /var/git/* areas as I do 
not want to "polute" the BOS (base OS) and/or other packages/packagers. 
Officially, according the Linux FHS-3.0 I sould be using /opt/aixtools 
as prefix.


FYI: my current build process is:
wget git-{version}.tar.xz
xz -dc git-{version}.tar.xz
cd git-{version}.tar.xz
mv Makefile Makefile.git #my build scripts only run configure when 
Makefile does not exist

./configure ...
ln Makefile.git Makefile
make

I am amazed, as it rarely happens (maybe git is my first encounter) - 
that configure does not create a Makefile. This also complicates 
building git "out of tree".


Junio: I've barked up that particular tree before in
https://public-inbox.org/git/87a7x3kmh5@evledraar.gmail.com/ and I
won't bore you all by repeating myself, except to say that this is yet
another case where I wish we had a hard dependency on some way of doing
checks via compiled code in our build system.
For AIX: again - the determination is simple. If _AIX is set to 1 then 
use BigEndian, or, use:

michael@x071:[/home/michael]uname
AIX
 i.e., something like:
$(uname) == "AIX" && BigEndian=1

Re: Is detecting endianness at compile-time unworkable?

[Michael Felt]

I have just replied to 
https://github.com/cr-marcstevens/sha1collisiondetection/pull/42


I checked a gcc compiler on AIX, and I have the defines for vac.

I do not have access yet to SLES or RHEL (or Ubuntu), just a "free 
Debian" on my Power6.


* my conclusions|recommendations:

a) AIX is always Big Endian, the define _AIX can be used to determine if AIX

b) POWER7 and earlier are always Big Endian

c) assuming lscpu is always available on Linux systems a command (in 
configure?) could be used:


root@x074:/usr/bin# lscpu | grep -i endian
Byte Order:    Big Endian

d) some linux systems (in any case latest versions of RHEL and SLES 
enterprise) should have a file named lparcfg in /proc 
(/proc/{powerppc|ppc64|ppc64le|ppc64el}/lparcfg - and it might be in 
that file. Need to get onto a (POWER8|POWER9) system to check.


Hope this helps:

details re: define of _AIX

root@x068:[/data/httpd/gcc]gcc -dM -E - < /dev/null | grep AIX | head -1
#define _AIX 1

michael@x071:[/home/michael]/usr/bin/grep -p DEFLT: 
/etc/vac.cfg.[567][123] | grep options\
    options   = 
-D_AIX,-D_AIX32,-D_AIX41,-D_AIX43,-D_AIX50,-D_AIX51,-D_AIX52,-D_AIX53,-D_IBMR2,-D_POWER
    options   = 
-D_AIX,-D_AIX32,-D_AIX41,-D_AIX43,-D_AIX50,-D_AIX51,-D_AIX52,-D_AIX53,-D_AIX61,-D_IBMR2,-D_POWER
    options   = 
-D_AIX,-D_AIX32,-D_AIX41,-D_AIX43,-D_AIX50,-D_AIX51,-D_AIX52,-D_AIX53,-D_AIX61,-D_AIX71,-D_IBMR2,-D_POWER
    options   = 
-D_AIX,-D_AIX32,-D_AIX41,-D_AIX43,-D_AIX50,-D_AIX51,-D_AIX52,-D_AIX53,-D_AIX61,-D_AIX71,-D_AIX72,-D_IBMR2,-D_POWER


michael@x071:[/home/michael]ls /etc/vac.cfg.[567][123]
/etc/vac.cfg.53  /etc/vac.cfg.61  /etc/vac.cfg.71  /etc/vac.cfg.72


On 7/30/2018 8:39 PM, Daniel Shumow wrote:
The change was definitely made for performance. Removing the if 
statements, conditioned upon endianess was an approx 10% improvement, 
which was very important to getting this library accepted into git.


Thanks,
Dan


On Mon, Jul 30, 2018 at 11:32 AM, Junio C Hamano > wrote:


Junio C Hamano mailto:gits...@pobox.com>> writes:

> Ævar Arnfjörð Bjarmason mailto:ava...@gmail.com>> writes:
>
>> And, as an aside, the reason we can't easily make it better
ourselves is
>> because the build process for git.git doesn't have a facility
to run
>> code to detect this type of stuff (the configure script is always
>> optional). So we can't just run this test ourselves.
>
> It won't help those who cross-compile anyway.  I thought we declared
> "we make a reasonable effort to guess the target endianness from the
> system header by inspecting usual macros, but will not aim to cover
> every system on the planet---instead there is a knob to tweak it for
> those on exotic platforms" last time we discussed this?

Well, having said all that, I do not think I personally mind if
./configure learned to include a "compile small program and run it
to determine byte order on the build machine" as part of "we make a
reasonable effort" as long as it cleanly excludes cross building
case (and the result is made overridable just in case we misdetect
the "cross-ness" of the build).

Re: Git For Aix 6 and 7

[Michael Felt]

I have git on my portal www.aixtools.net (See
http://www.aixtools.net/index.php/git)

These are installp packages - and what you install, you can uninstall.

There are some dependencies (e.g., python, gnu grep, and a few
others). Can't say I use it daily, but I do use it weekly.

For additional questions or issues with this packaging - please post
on http://forums.rootvg.net/aixtools - I see that a lot sooner than
any email (on gmail).

HTH,
Michael

On Thu, Jan 18, 2018 at 3:47 PM, Ævar Arnfjörð Bjarmason
 wrote:
>
> On Thu, Jan 18 2018, raikrishna jotted:
>
>> Hi Team,
>>
>> I have an urgent requirement to install Git client for Aix 6 and 7, could
>> you please help send me or navigate me to the correct url.
>> My present infrastructure comprise of Aix and Linux servers , I am
>> successfully using Git on Linux however I am struggling to find correct
>> package for AIX platform.
>>
>> Appreciate your quick response.
>
> Hi raikrishna. The git-packagers list is a rather small list so perhaps
> someone on the general git list (CC'd) knows the answer to this.
>
> I'm not aware of anyone providing binary git packages for AIX, but I
> don't use it so maybe they exist.
>
> The last mention on the mailing list I could find of someone packaging
> it was this from Michael Felt's (CC'd)
> https://public-inbox.org/git/canvxnixkbakgjm+nz0cyyctoeyp23kd8s4yxsquosauahjs...@mail.gmail.com/
>
> The last AIX-related patch to git is actually mine, but I haven't logged
> into an AIX box in over a decade, see
> https://github.com/chef/omnibus-software/commit/e247e36761#diff-3df898345d670979b74acc0bf71d8c47
>
> So it looks like there's a chef build recipe for it, maybe that's
> something you can use?
>
> I would not be surprised if building git on AIX, particularly with a
> non-GNU toolchain, fails in all sorts of interesting ways. People here
> on the list would be happy to help you work through those failures,
> we're keen to port git to whatever we can get our hands on, but these
> platforms experience quite a bit of bitrot.

Re: build issues on AIX - aka non-GNU environment, no gnu grep, no gcc

[Michael Felt]

Try again, reply all this time...


On 12-Sep-16 22:24, Junio C Hamano wrote:

Michael Felt <aixto...@gmail.com> writes:


I had a couple of issues when packaging git for AIX
a) option -Wall by default - works fine with gcc I am sure, but not so
well when gcc is not your compiler

That is expected.  As you said, it is merely "by default" and there
are mechanisms like config.mak provided for people to override them.


b) needs a special (GNU) grep argument (-a from memory). This I
resolved by downloading and packaging GNU grep. However, I hope this
has not introduced a new dependency.

Read the Makefile and find SANE_TEXT_GREP, perhaps?
Ah - read the manual heh?. Generally, I just download, unpack, and run 
configure.

I do not mind needing an extra tool - as long as it is only for the build.

And, since you have something about it in your (still unread) install 
Notes - it has not slipped in by accident.


Thanks for helping to spread Git to minority platforms.

The link is: http://www.aixtools.net/index.php/git
I do not use it much (I prefer to package "distributed" src tarballs, 
but when testing a potential fix - git can be extremely useful!


All in all, thanks for an easy to port to a "minority" platform!