[
https://issues.apache.org/jira/browse/IMPALA-12342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17757110#comment-17757110
]
ASF subversion and git services commented on IMPALA-12342:
----------------------------------------------------------
Commit a34f7ce63299c72ef45a99b01bb4e80210befbff in impala's branch
refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=a34f7ce63 ]
IMPALA-12342: Erasure coding build fails on loading iceberg_lineitem_multiblock
Previous to this patch we tried to load table
iceberg_lineitem_multiblock with HDFS block size 524288. This failed
in builds that use HDFS erasure coding which requires block size at
least 1048576.
This patch increases the block size to 1048576. This also triggers
the bug that was fixed by IMPALA-12327. But to have more tests with
multiblock tables this patch also adds table iceberg_lineitem_sixblocks
and few tests with different MT_DOP settings.
Testing:
* tested in build with HDFS EC
Change-Id: Iad15a335407c12578eb822bb1cb4450647502e50
Reviewed-on: http://gerrit.cloudera.org:8080/20359
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Erasure coding build fails on loading iceberg_lineitem_multiblock
> -----------------------------------------------------------------
>
> Key: IMPALA-12342
> URL: https://issues.apache.org/jira/browse/IMPALA-12342
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 4.2.0
> Environment: ERASURE_CODING=true
> Reporter: Peter Rozsa
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: test-failure
>
> Erasure coding build fails at statement generation:
>
> {code:java}
> Traceback (most recent call last):
> File "/home/impdev/impala/testdata/bin/generate-schema-statements.py", line
> 935, in <module>
> convert_orc_to_full_acid)
> File "/home/impdev/impala/testdata/bin/generate-schema-statements.py", line
> 678, in generate_statements
> insert = eval_section(section['DEPENDENT_LOAD'])
> File "/home/impdev/impala/testdata/bin/generate-schema-statements.py", line
> 612, in eval_section
> assert p.returncode == 0
> AssertionError{code}
>
> Error message:
>
> {code:java}
> put: Specified block size (524288) is less than the cell size (1048576) of
> the erasure coding policy (ErasureCodingPolicy=[Name=RS-3-2-1024k,
> Schema=[ECSchema=[Codec=rs, numDataUnits=3, numParityUnits=2]],
> CellSize=1048576, Id=2]).
> put: Specified block size (524288) is less than the cell size (1048576) of
> the erasure coding policy (ErasureCodingPolicy=[Name=RS-3-2-1024k,
> Schema=[ECSchema=[Codec=rs, numDataUnits=3, numParityUnits=2]],
> CellSize=1048576, Id=2]).
> put: Specified block size (524288) is less than the cell size (1048576) of
> the erasure coding policy (ErasureCodingPolicy=[Name=RS-3-2-1024k,
> Schema=[ECSchema=[Codec=rs, numDataUnits=3, numParityUnits=2]],
> CellSize=1048576, Id=2]).
> put: Specified block size (524288) is less than the cell size (1048576) of
> the erasure coding policy (ErasureCodingPolicy=[Name=RS-3-2-1024k,
> Schema=[ECSchema=[Codec=rs, numDataUnits=3, numParityUnits=2]],
> CellSize=1048576, Id=2]).
> put: Specified block size (524288) is less than the cell size (1048576) of
> the erasure coding policy (ErasureCodingPolicy=[Name=RS-3-2-1024k,
> Schema=[ECSchema=[Codec=rs, numDataUnits=3, numParityUnits=2]],
> CellSize=1048576, Id=2]).
> put: Specified block size (524288) is less than the cell size (1048576) of
> the erasure coding policy (ErasureCodingPolicy=[Name=RS-3-2-1024k,
> Schema=[ECSchema=[Codec=rs, numDataUnits=3, numParityUnits=2]],
> CellSize=1048576, Id=2]).
> put: Specified block size (524288) is less than the cell size (1048576) of
> the erasure coding policy (ErasureCodingPolicy=[Name=RS-3-2-1024k,
> Schema=[ECSchema=[Codec=rs, numDataUnits=3, numParityUnits=2]],
> CellSize=1048576, Id=2]).
> put: Specified block size (524288) is less than the cell size (1048576) of
> the erasure coding policy (ErasureCodingPolicy=[Name=RS-3-2-1024k,
> Schema=[ECSchema=[Codec=rs, numDataUnits=3, numParityUnits=2]],
> CellSize=1048576, Id=2]).
> put: Specified block size (524288) is less than the cell size (1048576) of
> the erasure coding policy (ErasureCodingPolicy=[Name=RS-3-2-1024k,
> Schema=[ECSchema=[Codec=rs, numDataUnits=3, numParityUnits=2]],
> CellSize=1048576, Id=2]).
> put: Specified block size (524288) is less than the cell size (1048576) of
> the erasure coding policy (ErasureCodingPolicy=[Name=RS-3-2-1024k,
> Schema=[ECSchema=[Codec=rs, numDataUnits=3, numParityUnits=2]],
> CellSize=1048576, Id=2]).{code}
>
> IMPALA-12327 added a specific block size setting for copying files from
> iceberg_lineitem_multiblock. Without the blocksize setting, the put operation
> succeeds.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
