[
https://issues.apache.org/jira/browse/FLINK-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16671689#comment-16671689
]
ASF GitHub Bot commented on FLINK-10627:
----------------------------------------
kl0u commented on a change in pull request #6957: [FLINK-10627][E2E tests] Test
s3 output for streaming file sink
URL: https://github.com/apache/flink/pull/6957#discussion_r230065738
##########
File path: flink-end-to-end-tests/test-scripts/test_streaming_file_sink.sh
##########
@@ -47,10 +57,32 @@ function wait_for_restart {
# Arguments:
# None
# Returns:
-# None
+# sorted content of part files
###################################
function get_complete_result {
- find "${OUTPUT_PATH}" -type f \( -iname "part-*" \) -exec cat {} + | sort
-g
+ if [ "${OUT_TYPE}" == "s3" ]; then
+ rm -rf $OUTPUT_PATH; mkdir -p $OUTPUT_PATH
+ s3_get_by_prefix ${TEST_DATA_DIR} "${OUT}" "part-"
+ fi
+ find "${OUTPUT_PATH}" -type f \( -iname "part-*" \) -exec cat {} + | sort -g
+}
+
+###################################
+# Get total number of lines in part files.
+#
+# Globals:
+# OUT
+# Arguments:
+# None
+# Returns:
+# line number in part files
+###################################
+function get_number_of_lines {
Review comment:
I would rename it to `get_total_number_of_valid_lines`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Extend test_streaming_file_sink to test S3 writer
> -------------------------------------------------
>
> Key: FLINK-10627
> URL: https://issues.apache.org/jira/browse/FLINK-10627
> Project: Flink
> Issue Type: Sub-task
> Components: filesystem-connector, Tests
> Affects Versions: 1.7.0
> Reporter: Till Rohrmann
> Assignee: Andrey Zagrebin
> Priority: Critical
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> We should extend the {{test_streaming_file_sink.sh}} to test the new S3
> recoverable writer in an integrated way.
> The test should cover the fault recovery. Moreover it should check for
> resource cleanup.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
