[
https://issues.apache.org/jira/browse/IMPALA-13923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17939983#comment-17939983
]
Surya Hebbar edited comment on IMPALA-13923 at 4/1/25 9:07 AM:
---------------------------------------------------------------
Currently, the tests have passed with changes in
[https://gerrit.cloudera.org/#/c/22718/1|https://gerrit.cloudera.org/#/c/22718/],
but with respect to compression levels and IMPALA-10630 many small changes to
tests still need to be made and refactored.
{code:java}
Note: Google Test filter = DecompressorTest.*
[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from DecompressorTest
[ RUN ] DecompressorTest.Default
[ OK ] DecompressorTest.Default (24 ms)
[ RUN ] DecompressorTest.Snappy
[ OK ] DecompressorTest.Snappy (3 ms)
[ RUN ] DecompressorTest.LZ4
[ OK ] DecompressorTest.LZ4 (3 ms)
[ RUN ] DecompressorTest.Gzip
[ OK ] DecompressorTest.Gzip (556 ms)
[ RUN ] DecompressorTest.Deflate
[ OK ] DecompressorTest.Deflate (604 ms)
[ RUN ] DecompressorTest.Bzip
block 1: crc = 0x8c6d307b, combined CRC = 0x8c6d307b, size = 53248
too repetitive; using fallback sorting algorithm
final combined CRC = 0x8c6d307b
block 1: crc = 0x9221421d, combined CRC = 0x9221421d, size = 53247
too repetitive; using fallback sorting algorithm
final combined CRC = 0x9221421d
block 1: crc = 0x8f428572, combined CRC = 0x8f428572, size = 1024
final combined CRC = 0x8f428572
final combined CRC = 0x00000000
block 1: crc = 0x8c6d307b, combined CRC = 0x8c6d307b, size = 53248
too repetitive; using fallback sorting algorithm
final combined CRC = 0x8c6d307b
block 1: crc = 0x626f6967, combined CRC = 0x626f6967, size = 499983
too repetitive; using fallback sorting algorithm
block 2: crc = 0x095d78be, combined CRC = 0xcd83aa70, size = 262358
too repetitive; using fallback sorting algorithm
final combined CRC = 0xcd83aa70
final combined CRC = 0x00000000
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x88af49e6, combined CRC = 0xf65200d0, size = 48402
final combined CRC = 0xf65200d0
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x65e17597, combined CRC = 0x1b1c3ca1, size = 48189
final combined CRC = 0x1b1c3ca1
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0xc8c5599c, combined CRC = 0xb63810aa, size = 48197
final combined CRC = 0xb63810aa
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0xd9771205, combined CRC = 0xa78a5b33, size = 47845
final combined CRC = 0xa78a5b33
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x65335cc1, combined CRC = 0x1bce15f7, size = 47742
final combined CRC = 0x1bce15f7
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x871feffe, combined CRC = 0xf9e2a6c8, size = 48367
final combined CRC = 0xf9e2a6c8
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x8c9f1774, combined CRC = 0xf2625e42, size = 47788
final combined CRC = 0xf2625e42
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x85d40238, combined CRC = 0xfb294b0e, size = 48320
final combined CRC = 0xfb294b0e
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0xba2143a1, combined CRC = 0xc4dc0a97, size = 48203
final combined CRC = 0xc4dc0a97
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x5f5e9749, combined CRC = 0x21a3de7f, size = 48060
final combined CRC = 0x21a3de7f
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x522c81c9, combined CRC = 0x2cd1c8ff, size = 47859
final combined CRC = 0x2cd1c8ff
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x56c43404, combined CRC = 0x28397d32, size = 47925
final combined CRC = 0x28397d32
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x69e3f994, combined CRC = 0x171eb0a2, size = 47938
final combined CRC = 0x171eb0a2
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x121b27e2, combined CRC = 0x6ce66ed4, size = 47722
final combined CRC = 0x6ce66ed4
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0xb5ac5923, combined CRC = 0xcb511015, size = 47959
final combined CRC = 0xcb511015
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x2bea0534, combined CRC = 0x55174c02, size = 48291
final combined CRC = 0x55174c02
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x72ed70c8, combined CRC = 0x0c1039fe, size = 48133
final combined CRC = 0x0c1039fe
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0xabf6635c, combined CRC = 0xd50b2a6a, size = 48220
final combined CRC = 0xd50b2a6a
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x569ccd19, combined CRC = 0x2861842f, size = 48152
final combined CRC = 0x2861842f
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x559db661, combined CRC = 0x2b60ff57, size = 47978
final combined CRC = 0x2b60ff57
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x8d97cf05, combined CRC = 0xf36a8633, size = 47659
final combined CRC = 0xf36a8633
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0xbf612e26, combined CRC = 0xc19c6710, size = 47646
final combined CRC = 0xc19c6710
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x2cd1a223, combined CRC = 0x522ceb15, size = 47991
final combined CRC = 0x522ceb15
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0xf4af9eb2, combined CRC = 0x8a52d784, size = 48004
final combined CRC = 0x8a52d784
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x1e35b363, combined CRC = 0x60c8fa55, size = 48531
final combined CRC = 0x60c8fa55
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x85ea2a0d, combined CRC = 0xfb17633b, size = 48233
final combined CRC = 0xfb17633b
[ OK ] DecompressorTest.Bzip (4274 ms)
[ RUN ] DecompressorTest.SnappyBlocked
[ OK ] DecompressorTest.SnappyBlocked (1 ms)
[ RUN ] DecompressorTest.Impala1506
[ OK ] DecompressorTest.Impala1506 (0 ms)
[ RUN ] DecompressorTest.Impala5250
[ OK ] DecompressorTest.Impala5250 (0 ms)
[ RUN ] DecompressorTest.LZ4Huge
[ OK ] DecompressorTest.LZ4Huge (25219 ms)
[ RUN ] DecompressorTest.ZSTD
[ OK ] DecompressorTest.ZSTD (18 ms)
[ RUN ] DecompressorTest.ZSTDHuge
[ OK ] DecompressorTest.ZSTDHuge (1255 ms)
[ RUN ] DecompressorTest.LZ4HadoopCompat
[ OK ] DecompressorTest.LZ4HadoopCompat (1 ms)
[ RUN ] DecompressorTest.LZ4Blocked
[ OK ] DecompressorTest.LZ4Blocked (0 ms)
[----------] 14 tests from DecompressorTest (31966 ms total)[----------] Global
test environment tear-down
[==========] 14 tests from 1 test suite ran. (31966 ms total)
[ PASSED ] 14 tests.
{code}
was (Author: JIRAUSER299620):
Currently, the tests have passed with changes in
[https://gerrit.cloudera.org/#/c/22718/1|https://gerrit.cloudera.org/#/c/22718/],
but with respect to compression levels and IMPALA-10630 many tests still need
to be added and refactored.
{code:java}
Note: Google Test filter = DecompressorTest.*
[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from DecompressorTest
[ RUN ] DecompressorTest.Default
[ OK ] DecompressorTest.Default (24 ms)
[ RUN ] DecompressorTest.Snappy
[ OK ] DecompressorTest.Snappy (3 ms)
[ RUN ] DecompressorTest.LZ4
[ OK ] DecompressorTest.LZ4 (3 ms)
[ RUN ] DecompressorTest.Gzip
[ OK ] DecompressorTest.Gzip (556 ms)
[ RUN ] DecompressorTest.Deflate
[ OK ] DecompressorTest.Deflate (604 ms)
[ RUN ] DecompressorTest.Bzip
block 1: crc = 0x8c6d307b, combined CRC = 0x8c6d307b, size = 53248
too repetitive; using fallback sorting algorithm
final combined CRC = 0x8c6d307b
block 1: crc = 0x9221421d, combined CRC = 0x9221421d, size = 53247
too repetitive; using fallback sorting algorithm
final combined CRC = 0x9221421d
block 1: crc = 0x8f428572, combined CRC = 0x8f428572, size = 1024
final combined CRC = 0x8f428572
final combined CRC = 0x00000000
block 1: crc = 0x8c6d307b, combined CRC = 0x8c6d307b, size = 53248
too repetitive; using fallback sorting algorithm
final combined CRC = 0x8c6d307b
block 1: crc = 0x626f6967, combined CRC = 0x626f6967, size = 499983
too repetitive; using fallback sorting algorithm
block 2: crc = 0x095d78be, combined CRC = 0xcd83aa70, size = 262358
too repetitive; using fallback sorting algorithm
final combined CRC = 0xcd83aa70
final combined CRC = 0x00000000
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x88af49e6, combined CRC = 0xf65200d0, size = 48402
final combined CRC = 0xf65200d0
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x65e17597, combined CRC = 0x1b1c3ca1, size = 48189
final combined CRC = 0x1b1c3ca1
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0xc8c5599c, combined CRC = 0xb63810aa, size = 48197
final combined CRC = 0xb63810aa
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0xd9771205, combined CRC = 0xa78a5b33, size = 47845
final combined CRC = 0xa78a5b33
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x65335cc1, combined CRC = 0x1bce15f7, size = 47742
final combined CRC = 0x1bce15f7
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x871feffe, combined CRC = 0xf9e2a6c8, size = 48367
final combined CRC = 0xf9e2a6c8
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x8c9f1774, combined CRC = 0xf2625e42, size = 47788
final combined CRC = 0xf2625e42
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x85d40238, combined CRC = 0xfb294b0e, size = 48320
final combined CRC = 0xfb294b0e
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0xba2143a1, combined CRC = 0xc4dc0a97, size = 48203
final combined CRC = 0xc4dc0a97
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x5f5e9749, combined CRC = 0x21a3de7f, size = 48060
final combined CRC = 0x21a3de7f
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x522c81c9, combined CRC = 0x2cd1c8ff, size = 47859
final combined CRC = 0x2cd1c8ff
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x56c43404, combined CRC = 0x28397d32, size = 47925
final combined CRC = 0x28397d32
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x69e3f994, combined CRC = 0x171eb0a2, size = 47938
final combined CRC = 0x171eb0a2
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x121b27e2, combined CRC = 0x6ce66ed4, size = 47722
final combined CRC = 0x6ce66ed4
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0xb5ac5923, combined CRC = 0xcb511015, size = 47959
final combined CRC = 0xcb511015
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x2bea0534, combined CRC = 0x55174c02, size = 48291
final combined CRC = 0x55174c02
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x72ed70c8, combined CRC = 0x0c1039fe, size = 48133
final combined CRC = 0x0c1039fe
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0xabf6635c, combined CRC = 0xd50b2a6a, size = 48220
final combined CRC = 0xd50b2a6a
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x569ccd19, combined CRC = 0x2861842f, size = 48152
final combined CRC = 0x2861842f
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x559db661, combined CRC = 0x2b60ff57, size = 47978
final combined CRC = 0x2b60ff57
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x8d97cf05, combined CRC = 0xf36a8633, size = 47659
final combined CRC = 0xf36a8633
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0xbf612e26, combined CRC = 0xc19c6710, size = 47646
final combined CRC = 0xc19c6710
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x2cd1a223, combined CRC = 0x522ceb15, size = 47991
final combined CRC = 0x522ceb15
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0xf4af9eb2, combined CRC = 0x8a52d784, size = 48004
final combined CRC = 0x8a52d784
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x1e35b363, combined CRC = 0x60c8fa55, size = 48531
final combined CRC = 0x60c8fa55
block 1: crc = 0xb5321fd0, combined CRC = 0xb5321fd0, size = 499981
block 2: crc = 0x551a9b3a, combined CRC = 0x3f7ea49b, size = 499981
block 3: crc = 0x85ea2a0d, combined CRC = 0xfb17633b, size = 48233
final combined CRC = 0xfb17633b
[ OK ] DecompressorTest.Bzip (4274 ms)
[ RUN ] DecompressorTest.SnappyBlocked
[ OK ] DecompressorTest.SnappyBlocked (1 ms)
[ RUN ] DecompressorTest.Impala1506
[ OK ] DecompressorTest.Impala1506 (0 ms)
[ RUN ] DecompressorTest.Impala5250
[ OK ] DecompressorTest.Impala5250 (0 ms)
[ RUN ] DecompressorTest.LZ4Huge
[ OK ] DecompressorTest.LZ4Huge (25219 ms)
[ RUN ] DecompressorTest.ZSTD
[ OK ] DecompressorTest.ZSTD (18 ms)
[ RUN ] DecompressorTest.ZSTDHuge
[ OK ] DecompressorTest.ZSTDHuge (1255 ms)
[ RUN ] DecompressorTest.LZ4HadoopCompat
[ OK ] DecompressorTest.LZ4HadoopCompat (1 ms)
[ RUN ] DecompressorTest.LZ4Blocked
[ OK ] DecompressorTest.LZ4Blocked (0 ms)
[----------] 14 tests from DecompressorTest (31966 ms total)[----------] Global
test environment tear-down
[==========] 14 tests from 1 test suite ran. (31966 ms total)
[ PASSED ] 14 tests.
{code}
> Support compression levels for GZIP
> -----------------------------------
>
> Key: IMPALA-13923
> URL: https://issues.apache.org/jira/browse/IMPALA-13923
> Project: IMPALA
> Issue Type: Improvement
> Reporter: Surya Hebbar
> Assignee: Surya Hebbar
> Priority: Major
>
> Support compression levels for GZIP, while possibly refactoring existing
> tests and implementation of compression, while also possibly trying to
> support resolve, IMPALA-10630 for ZSTD in later changes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
