[
https://issues.apache.org/jira/browse/FLINK-11990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16798106#comment-16798106
]
Yu Li edited comment on FLINK-11990 at 3/21/19 1:47 PM:
--------------------------------------------------------
Thanks for the double check [~aljoscha]. Just ran locally with {{skip}} ahead
in the command and confirmed 2.8.3 could also pass.
However, I think this {{skip}} may has given some misleading information since
it also ignores the error in the output of the test script. Allow me to repeat
the output with hadoop 2.8.3:
{noformat}
Waiting for job (905ae10bae4b99031e724b9c29f0ca7b) to reach terminal state
FINISHED ...
Truncating buckets
Truncating to
{noformat}
Where the relative lines of {{test_streaming_bucketing.sh}} script are:
{noformat}
LOG_LINES=$(grep -rnw $FLINK_DIR/log -e 'Writing valid-length file')
# perform truncate on every line
echo "Truncating buckets"
while read -r LOG_LINE; do
PART=$(echo "$LOG_LINE" | awk '{ print $10 }' FS=" ")
LENGTH=$(echo "$LOG_LINE" | awk '{ print $15 }' FS=" ")
echo "Truncating $PART to $LENGTH"
dd if=$PART of="$PART.truncated" bs=$LENGTH count=1
rm $PART
mv "$PART.truncated" $PART
done <<< "$LOG_LINES"
{noformat}
So it means there's no message like "Writing valid-length file" in flink task
logs. And I think this indicates some problem happens. Wdyt? Thanks.
was (Author: carp84):
Thanks for the double check [~aljoscha]. Just ran locally with {{skip}} ahead
in the command and confirmed 2.8.3 could also pass.
However, I think this {{skip}} may has given some misleading information since
it also ignores the error in the output of the test script. Allow me to repeat
the output with hadoop 2.8.3:
{noformat}
Waiting for job (905ae10bae4b99031e724b9c29f0ca7b) to reach terminal state
FINISHED ...
Truncating buckets
Truncating to
{noformat}
Where the relative lines of {{test_streaming_bucketing.sh}} script are:
{noformat}
LOG_LINES=$(grep -rnw $FLINK_DIR/log -e 'Writing valid-length file')
# perform truncate on every line
echo "Truncating buckets"
while read -r LOG_LINE; do
PART=$(echo "$LOG_LINE" | awk '{ print $10 }' FS=" ")
LENGTH=$(echo "$LOG_LINE" | awk '{ print $15 }' FS=" ")
echo "Truncating $PART to $LENGTH"
dd if=$PART of="$PART.truncated" bs=$LENGTH count=1
rm $PART
mv "$PART.truncated" $PART
done <<< "$LOG_LINES"
{noformat}
So it means there's no message like "Writing valid-length file" in logs of the
flink dist. And I think this indicates some problem happens. Wdyt? Thanks.
> Streaming bucketing end-to-end test fail with hadoop 2.8
> --------------------------------------------------------
>
> Key: FLINK-11990
> URL: https://issues.apache.org/jira/browse/FLINK-11990
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hadoop Compatibility
> Reporter: Yu Li
> Priority: Critical
>
> As titled, running the {{test_streaming_bucketing.sh}} case with hadoop 2.8
> bundles always fail, while running with 2.6 bundles could pass.
> Command to run the case:
> {{FLINK_DIR=<flink dir> flink-end-to-end-tests/run-single-test.sh
> test-scripts/test_streaming_bucketing.sh skip_check_exceptions}}
> The output with hadoop 2.8
> [bundle|https://repository.apache.org/content/repositories/orgapacheflink-1213/org/apache/flink/flink-shaded-hadoop2-uber/2.8.3-1.8.0/flink-shaded-hadoop2-uber-2.8.3-1.8.0.jar]
> or [dist|http://archive.apache.org/dist/hadoop/core/hadoop-2.8.5]:
> {noformat}
> Starting taskexecutor daemon on host z05f06378.sqa.zth.
> Waiting for job (905ae10bae4b99031e724b9c29f0ca7b) to reach terminal state
> FINISHED ...
> Truncating buckets
> Truncating to
> {noformat}
> The output of the success run with hadoop 2.6
> [bundle|https://repository.apache.org/content/repositories/orgapacheflink-1213/org/apache/flink/flink-shaded-hadoop2-uber/2.6.5-1.8.0/flink-shaded-hadoop2-uber-2.6.5-1.8.0.jar]
> or [dist|http://archive.apache.org/dist/hadoop/core/hadoop-2.6.5]:
> {noformat}
> Truncating
> /home/jueding.ly/flink_rc_check/flink-1.8.0-src/flink-end-to-end-tests/test-scripts/temp-test-directory-06210353709/out/result3/part-3-0
> to 51250
> 1+0 records in
> 1+0 records out
> 51250 bytes (51 kB) copied, 0.000377998 s, 136 MB/s
> Truncating
> /home/jueding.ly/flink_rc_check/flink-1.8.0-src/flink-end-to-end-tests/test-scripts/temp-test-directory-06210353709/out/result7/part-3-0
> to 51250
> 1+0 records in
> 1+0 records out
> 51250 bytes (51 kB) copied, 0.00033118 s, 155 MB/s
> pass Bucketing Sink
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
