Re: [PATCH v2 00/13] Base SHA-256 implementation

2018-10-16 Thread brian m. carlson
On Tue, Oct 16, 2018 at 01:01:07PM +0900, Junio C Hamano wrote:
> "brian m. carlson"  writes:
> 
> >  t/t0014-hash.sh   |  54 
> >  create mode 100755 t/t0014-hash.sh
> 
> If I am not mistaken, 0014 is already used by another topic in
> flight, and will cause test-lint failure on 'pu'.

This series was written a while back.  I'll rename it.  Feel free to
wait for v3 before picking it up.
-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204


signature.asc
Description: PGP signature


Re: [PATCH v2 00/13] Base SHA-256 implementation

2018-10-16 Thread Duy Nguyen
On Mon, Oct 15, 2018 at 4:21 AM brian m. carlson
 wrote:
>
> This series provides an actual SHA-256 implementation and wires it up,
> along with some housekeeping patches to make it suitable for testing.
>
> New in this version is a patch which reverts the change to limit hashcmp
> to 20 bytes.  I've taken care to permit the compiler to inline as much
> as possible for efficiency, but it would be helpful to see what the
> performance impact of these changes is on real-life workflows, or with
> MSVC and other non-GCC and non-clang compilers.  The resulting change is
> uglier and more duplicative than I wanted, but oh, well.
>
> I didn't attempt to pull in the full complement of code from libtomcrypt
> to try to show the changes made, since that would have involved
> importing a significant quantity of code in order to make things work.
>
> I realize the increase to GIT_MAX_HEXSZ will result in an increase in
> memory usage, but we're going to need to do it at some point, and the
> sooner the code is in the codebase, the sooner people can play around
> with it and test it.
>
> This piece should be the final piece of preparatory work required for a
> fully functioning SHA-256-only Git.  Additional work should be able to
> come into the testsuite and codebase without needing to build on work in
> any series after this one.

Thanks again for keeping working on this. I'm still sick and can't
really do a deep review. With that in mind, the patches look good.
-- 
Duy


Re: [PATCH v2 00/13] Base SHA-256 implementation

2018-10-15 Thread Junio C Hamano
"brian m. carlson"  writes:

>  t/t0014-hash.sh   |  54 
>  create mode 100755 t/t0014-hash.sh

If I am not mistaken, 0014 is already used by another topic in
flight, and will cause test-lint failure on 'pu'.



Re: [PATCH v2 00/13] Base SHA-256 implementation

2018-10-15 Thread Junio C Hamano
"brian m. carlson"  writes:

> I realize the increase to GIT_MAX_HEXSZ will result in an increase in
> memory usage, but we're going to need to do it at some point, and the
> sooner the code is in the codebase, the sooner people can play around
> with it and test it.

Thanks.


[PATCH v2 00/13] Base SHA-256 implementation

2018-10-14 Thread brian m. carlson
This series provides an actual SHA-256 implementation and wires it up,
along with some housekeeping patches to make it suitable for testing.

New in this version is a patch which reverts the change to limit hashcmp
to 20 bytes.  I've taken care to permit the compiler to inline as much
as possible for efficiency, but it would be helpful to see what the
performance impact of these changes is on real-life workflows, or with
MSVC and other non-GCC and non-clang compilers.  The resulting change is
uglier and more duplicative than I wanted, but oh, well.

I didn't attempt to pull in the full complement of code from libtomcrypt
to try to show the changes made, since that would have involved
importing a significant quantity of code in order to make things work.

I realize the increase to GIT_MAX_HEXSZ will result in an increase in
memory usage, but we're going to need to do it at some point, and the
sooner the code is in the codebase, the sooner people can play around
with it and test it.

This piece should be the final piece of preparatory work required for a
fully functioning SHA-256-only Git.  Additional work should be able to
come into the testsuite and codebase without needing to build on work in
any series after this one.

v1, which was RFC, can be seen at
https://public-inbox.org/git/20180829005857.980820-1-sand...@crustytoothpaste.net/.

Changes from v1:
* Add a hash_to_hex function mirroring sha1_to_hex, but for
  the_hash_algo.
* Strip commit message explanation about why we chose SHA-256.
* Rebase on master
* Strip leading whitespace from commit message.
* Improve commit-graph patch to cover new code added since v1.
* Be more honest about the scope of work involved in porting the SHA-256
  implementation out of libtomcrypt.
* Revert change to limit hashcmp to 20 bytes.

brian m. carlson (13):
  sha1-file: rename algorithm to "sha1"
  sha1-file: provide functions to look up hash algorithms
  hex: introduce functions to print arbitrary hashes
  cache: make hashcmp and hasheq work with larger hashes
  t: add basic tests for our SHA-1 implementation
  t: make the sha1 test-tool helper generic
  sha1-file: add a constant for hash block size
  t/helper: add a test helper to compute hash speed
  commit-graph: convert to using the_hash_algo
  Add a base implementation of SHA-256 support
  sha256: add an SHA-256 implementation using libgcrypt
  hash: add an SHA-256 implementation using OpenSSL
  commit-graph: specify OID version for SHA-256

 Makefile  |  22 
 cache.h   |  51 +---
 commit-graph.c|  40 +++---
 hash.h|  41 +-
 hex.c |  37 --
 sha1-file.c   |  70 +-
 sha256/block/sha256.c | 180 ++
 sha256/block/sha256.h |  26 
 sha256/gcrypt.h   |  30 +
 t/helper/test-hash-speed.c|  61 +
 t/helper/{test-sha1.c => test-hash.c} |  19 +--
 t/helper/test-sha1.c  |  52 +---
 t/helper/test-sha256.c|   7 +
 t/helper/test-tool.c  |   2 +
 t/helper/test-tool.h  |   4 +
 t/t0014-hash.sh   |  54 
 16 files changed, 592 insertions(+), 104 deletions(-)
 create mode 100644 sha256/block/sha256.c
 create mode 100644 sha256/block/sha256.h
 create mode 100644 sha256/gcrypt.h
 create mode 100644 t/helper/test-hash-speed.c
 copy t/helper/{test-sha1.c => test-hash.c} (66%)
 create mode 100644 t/helper/test-sha256.c
 create mode 100755 t/t0014-hash.sh

Range-diff against v1:
 1:  0a96c59452 =  1:  804ec2fd27 sha1-file: rename algorithm to "sha1"
 2:  65f9feba41 =  2:  5196e00b26 sha1-file: provide functions to look up hash 
algorithms
 3:  b6d960121d !  3:  5873510d0a hex: introduce functions to print arbitrary 
hashes
@@ -13,9 +13,12 @@
 it.
 
 We use the variant taking the algorithm structure pointer as the
-internal variant, since in the future we'll want to replace sha1_to_hex
-with a hash_to_hex that handles the_hash_algo, and taking an algorithm
-pointer is the easiest way to handle all of the variants in use.
+internal variant, since taking an algorithm pointer is the easiest way
+to handle all of the variants in use.
+
+Note that we maintain these functions because there are hashes which
+must change based on the hash algorithm in use but are not object IDs
+(such as pack checksums).
 
 Signed-off-by: brian m. carlson 
 
@@ -47,6 +50,7 @@
 +char *oid_to_hex_r(char *out, const struct object_id *oid);
 +char *hash_to_hex_algo(const unsigned char *hash, int algo);  /* 
static buffer result! */
 +char *sha1_to_hex(const unsigned char *sha1); /* same 
static buffer */
++char *hash_to_hex(const