[
https://issues.apache.org/jira/browse/FLINK-11990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16798221#comment-16798221
]
Aljoscha Krettek commented on FLINK-11990:
------------------------------------------
The reason for this is that truncate support was first introduced in Hadoop
2.7.0 (HDFS-3107). For versions after 2.7.0 the {{BucketingSink}} will not
write {{.valid-length}} files but directly truncate the file. The test
simulates this behaviour by manually enumerating the {{.valid-length}} files by
looking at the log entries.
For Hadoop 2.8.x {{LOG_LINES}} is empty. It seems bash is a bit strange here
and will have one iteration of the loop with an empty string, that's why you
see {{Truncating to}}, i.e. its truncating nothing. That's also why you see
the output from {{mv}} and {{rm}}.
> Streaming bucketing end-to-end test fail with hadoop 2.8
> --------------------------------------------------------
>
> Key: FLINK-11990
> URL: https://issues.apache.org/jira/browse/FLINK-11990
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hadoop Compatibility
> Reporter: Yu Li
> Priority: Critical
>
> As titled, running the {{test_streaming_bucketing.sh}} case with hadoop 2.8
> bundles always fail, while running with 2.6 bundles could pass.
> Command to run the case:
> {{FLINK_DIR=<flink dir> flink-end-to-end-tests/run-single-test.sh
> test-scripts/test_streaming_bucketing.sh skip_check_exceptions}}
> The output with hadoop 2.8
> [bundle|https://repository.apache.org/content/repositories/orgapacheflink-1213/org/apache/flink/flink-shaded-hadoop2-uber/2.8.3-1.8.0/flink-shaded-hadoop2-uber-2.8.3-1.8.0.jar]
> or [dist|http://archive.apache.org/dist/hadoop/core/hadoop-2.8.5]:
> {noformat}
> Starting taskexecutor daemon on host z05f06378.sqa.zth.
> Waiting for job (905ae10bae4b99031e724b9c29f0ca7b) to reach terminal state
> FINISHED ...
> Truncating buckets
> Truncating to
> {noformat}
> The output of the success run with hadoop 2.6
> [bundle|https://repository.apache.org/content/repositories/orgapacheflink-1213/org/apache/flink/flink-shaded-hadoop2-uber/2.6.5-1.8.0/flink-shaded-hadoop2-uber-2.6.5-1.8.0.jar]
> or [dist|http://archive.apache.org/dist/hadoop/core/hadoop-2.6.5]:
> {noformat}
> Truncating
> /home/jueding.ly/flink_rc_check/flink-1.8.0-src/flink-end-to-end-tests/test-scripts/temp-test-directory-06210353709/out/result3/part-3-0
> to 51250
> 1+0 records in
> 1+0 records out
> 51250 bytes (51 kB) copied, 0.000377998 s, 136 MB/s
> Truncating
> /home/jueding.ly/flink_rc_check/flink-1.8.0-src/flink-end-to-end-tests/test-scripts/temp-test-directory-06210353709/out/result7/part-3-0
> to 51250
> 1+0 records in
> 1+0 records out
> 51250 bytes (51 kB) copied, 0.00033118 s, 155 MB/s
> pass Bucketing Sink
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
