rustyconover commented on issue #8530: URL: https://github.com/apache/iceberg/issues/8530#issuecomment-1712493430
Hi Fokko, This seems to be a problem with integer widths in decoder_basic.c. On my machines unsigned longs are typically 64 bits. But not on windows. Reading: https://learn.microsoft.com/en-us/cpp/cpp/data-type-ranges?view=msvc-170 From that document an unsigned long is 32 bits. So we should use stdint.h and uint64_t. I can send a PR. I’m glad I wrote those bounds checking tests. Rusty On Sat, Sep 9, 2023 at 06:47 Fokko Driesprong ***@***.***> wrote: > Now including the pyd files in the build, and that fixed it. > > Now there seems to be an issue with decoding the extreme values: > > ______________ test_read_int_custom_encode[<lambda>-4294967296] _______________ > > decoder_class = <function <lambda> at 0x00000151BBE3F9D0> > expected_value = 4294967296 > > @pytest.mark.parametrize( > "decoder_class, expected_value", > list(itertools.product(AVAILABLE_DECODERS, [0, -1, 2**32, -(2**32), (2**63 - 1), -(2**63)])), > ) > def test_read_int_custom_encode(decoder_class: CALLABLE_DECODER, expected_value: int) -> None: > encoded = zigzag_encode(expected_value) > mis = io.BytesIO(encoded) > decoder = decoder_class(mis) > decoded = decoder.read_int() > > assert decoded == expected_value, f"Decoded value does not match decoded={decoded} expected={expected_value}" > E AssertionError: Decoded value does not match decoded=0 expected=4294967296 > E assert 0 == 4294967296 > > D:\a\incubator-iceberg\incubator-iceberg\python\tests\avro\test_decoder.py:96: AssertionError > ______________ test_read_int_custom_encode[<lambda>--4294967296] ______________ > > decoder_class = <function <lambda> at 0x00000151BBE3F9D0> > expected_value = -4294967296 > > @pytest.mark.parametrize( > "decoder_class, expected_value", > list(itertools.product(AVAILABLE_DECODERS, [0, -1, 2**32, -(2**32), (2**63 - 1), -(2**63)])), > ) > def test_read_int_custom_encode(decoder_class: CALLABLE_DECODER, expected_value: int) -> None: > encoded = zigzag_encode(expected_value) > mis = io.BytesIO(encoded) > decoder = decoder_class(mis) > decoded = decoder.read_int() > > assert decoded == expected_value, f"Decoded value does not match decoded={decoded} expected={expected_value}" > E AssertionError: Decoded value does not match decoded=-2147483648 expected=-4294967296 > E assert -2147483648 == -4294967296 > > D:\a\incubator-iceberg\incubator-iceberg\python\tests\avro\test_decoder.py:96: AssertionError > __________ test_read_int_custom_encode[<lambda>-92233720368[547](https://github.com/Fokko/incubator-iceberg/actions/runs/6130435761/job/16639517931#step:7:555)75807] __________ > > decoder_class = <function <lambda> at 0x00000151BBE3F9D0> > expected_value = 9223372036854775807 > > @pytest.mark.parametrize( > "decoder_class, expected_value", > list(itertools.product(AVAILABLE_DECODERS, [0, -1, 2**32, -(2**32), (2**63 - 1), -(2**63)])), > ) > def test_read_int_custom_encode(decoder_class: CALLABLE_DECODER, expected_value: int) -> None: > encoded = zigzag_encode(expected_value) > mis = io.BytesIO(encoded) > decoder = decoder_class(mis) > decoded = decoder.read_int() > > assert decoded == expected_value, f"Decoded value does not match decoded={decoded} expected={expected_value}" > E AssertionError: Decoded value does not match decoded=2147483647 expected=9223372036854775807 > E assert 2147483647 == 9223372036854775807 > > D:\a\incubator-iceberg\incubator-iceberg\python\tests\avro\test_decoder.py:96: AssertionError > _________ test_read_int_custom_encode[<lambda>--9223372036854775808] __________ > > decoder_class = <function <lambda> at 0x00000151BBE3F9D0> > expected_value = -9223372036854775808 > > @pytest.mark.parametrize( > "decoder_class, expected_value", > list(itertools.product(AVAILABLE_DECODERS, [0, -1, 2**32, -(2**32), (2**63 - 1), -(2**63)])), > ) > def test_read_int_custom_encode(decoder_class: CALLABLE_DECODER, expected_value: int) -> None: > encoded = zigzag_encode(expected_value) > mis = io.BytesIO(encoded) > decoder = decoder_class(mis) > decoded = decoder.read_int() > > assert decoded == expected_value, f"Decoded value does not match decoded={decoded} expected={expected_value}" > E AssertionError: Decoded value does not match decoded=-2147483648 expected=-922337203685477[580](https://github.com/Fokko/incubator-iceberg/actions/runs/6130435761/job/16639517931#step:7:588)8 > E assert -2147483648 == -9223372036854775808 > > cc @rustyconover <https://github.com/rustyconover> > > — > Reply to this email directly, view it on GitHub > <https://github.com/apache/iceberg/issues/8530#issuecomment-1712482320>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAFSWJNVEL2AFMV7MISH7QDXZRCMPANCNFSM6AAAAAA4RI3VKY> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
