[
https://issues.apache.org/jira/browse/FLINK-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681084#comment-16681084
]
ASF GitHub Bot commented on FLINK-10801:
----------------------------------------
pnowojski closed pull request #7060: [FLINK-10801][e2e] Retry
verify_result_hash in elastichsearch-common
URL: https://github.com/apache/flink/pull/7060
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/flink-end-to-end-tests/test-scripts/common.sh
b/flink-end-to-end-tests/test-scripts/common.sh
index 36cc0df142c..6b8bd061f15 100644
--- a/flink-end-to-end-tests/test-scripts/common.sh
+++ b/flink-end-to-end-tests/test-scripts/common.sh
@@ -451,6 +451,16 @@ function cancel_job {
}
function check_result_hash {
+ local error_code=0
+ check_result_hash_no_exit "$@" || error_code=$?
+
+ if [ "$error_code" != "0" ]
+ then
+ exit $error_code
+ fi
+}
+
+function check_result_hash_no_exit {
local name=$1
local outfile_prefix=$2
local expected=$3
@@ -462,18 +472,19 @@ function check_result_hash {
actual=$(LC_ALL=C sort $outfile_prefix* | md5sum | awk '{print $1}')
else
echo "Neither 'md5' nor 'md5sum' binary available."
- exit 2
+ return 2
fi
if [[ "$actual" != "$expected" ]]
then
echo "FAIL $name: Output hash mismatch. Got $actual, expected $expected."
echo "head hexdump of actual:"
head $outfile_prefix* | hexdump -c
- exit 1
+ return 1
else
echo "pass $name"
# Output files are left behind in /tmp
fi
+ return 0
}
# This function starts the given number of task managers and monitors their
processes.
diff --git a/flink-end-to-end-tests/test-scripts/elasticsearch-common.sh
b/flink-end-to-end-tests/test-scripts/elasticsearch-common.sh
index 8bb9c42ab69..695905222f5 100644
--- a/flink-end-to-end-tests/test-scripts/elasticsearch-common.sh
+++ b/flink-end-to-end-tests/test-scripts/elasticsearch-common.sh
@@ -83,23 +83,34 @@ function verify_result_hash {
local name=$1
local index=$2
local numRecords=$3
- local hash=$4
+ local expectedHash=$4
- while : ; do
+ for i in {1..30}
+ do
+ local error_code=0
+
+ echo "Result verification attempt $i..."
curl "localhost:9200/${index}/_search?q=*&pretty" >
$TEST_DATA_DIR/es_output || true
- if [ -n "$(grep "\"total\" : $numRecords" $TEST_DATA_DIR/es_output)" ];
then
- break
- else
- echo "Waiting for Elasticsearch records ..."
+ # remove meta information
+ sed '2,9d' $TEST_DATA_DIR/es_output > $TEST_DATA_DIR/es_content
+
+ check_result_hash_no_exit "$name" $TEST_DATA_DIR/es_content
"$expectedHash" || error_code=$?
+
+ if [ "$error_code" != "0" ]
+ then
+ echo "Result verification attempt $i has failed"
sleep 1
+ else
+ break
fi
done
- # remove meta information
- sed '2,9d' $TEST_DATA_DIR/es_output > $TEST_DATA_DIR/es_content
-
- check_result_hash "$name" $TEST_DATA_DIR/es_content "$hash"
+ if [ "$error_code" != "0" ]
+ then
+ echo "All result verification attempts have failed"
+ exit $error_code
+ fi
}
function shutdown_elasticsearch_cluster {
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Fix sql client integrate elasticsearch connector test failure
> -------------------------------------------------------------
>
> Key: FLINK-10801
> URL: https://issues.apache.org/jira/browse/FLINK-10801
> Project: Flink
> Issue Type: Bug
> Components: E2E Tests
> Reporter: vinoyang
> Assignee: Piotr Nowojski
> Priority: Major
> Labels: pull-request-available
>
> It usually reports :
> {code:java}
> FAIL SQL Client Elasticsearch Upsert: Output hash mismatch. Got
> 6187222e109ee9222e6b2f117742070c, expected 982cb32908def9801e781381c1b8a8db.
> head hexdump of actual:
> 0000000 { \n " h i t s " : { \n
> 0000010 " t o t a l " : 3 , \n
> 0000020 " m a x _ s c o r e "
> 0000030 : 1 . 0 , \n " h i t s
> 0000040 " : [ \n { \n
> 0000050 " _ i n d e x " :
> 0000060 " m y _ u s e r s " , \n
> 0000070 " _ t y p e " : "
> 0000080 u s e r " , \n "
> 0000090 _ i d " : " 1 _ B o b "
> 00000a0 , \n " _ s c o r
> 00000b0 e " : 1 . 0 , \n
> 00000ba
> {code}
> the actual hash means :
> {code:java}
> {
> "hits" : {
> "total" : 3,
> "max_score" : 1.0,
> "hits" : [
> {
> "_index" : "my_users",
> "_type" : "user",
> "_id" : "1_Bob ",
> "_score" : 1.0,
> "_source" : {
> "user_id" : 1,
> "user_name" : "Bob ",
> "user_count" : 1
> }
> },
> {
> "_index" : "my_users",
> "_type" : "user",
> "_id" : "22_Alice",
> "_score" : 1.0,
> "_source" : {
> "user_id" : 22,
> "user_name" : "Alice",
> "user_count" : 1
> }
> },
> {
> "_index" : "my_users",
> "_type" : "user",
> "_id" : "42_Greg ",
> "_score" : 1.0,
> "_source" : {
> "user_id" : 42,
> "user_name" : "Greg ",
> "user_count" : 3
> }
> }
> ]
> }
> }
> {code}
> the expected hash code means :
> {code:java}
> {
> "hits" : {
> "total" : 3,
> "max_score" : 1.0,
> "hits" : [
> {
> "_index" : "my_users",
> "_type" : "user",
> "_id" : "1_Bob ",
> "_score" : 1.0,
> "_source" : {
> "user_id" : 1,
> "user_name" : "Bob ",
> "user_count" : 2
> }
> },
> {
> "_index" : "my_users",
> "_type" : "user",
> "_id" : "22_Alice",
> "_score" : 1.0,
> "_source" : {
> "user_id" : 22,
> "user_name" : "Alice",
> "user_count" : 1
> }
> },
> {
> "_index" : "my_users",
> "_type" : "user",
> "_id" : "42_Greg ",
> "_score" : 1.0,
> "_source" : {
> "user_id" : 42,
> "user_name" : "Greg ",
> "user_count" : 3
> }
> }
> ]
> }
> }
> {code}
> It seems that the user count for "Bob" is off by 1.
> The speculation is due to the premature acquisition of aggregated statistics
> from Elasticsearch.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
