Will Berkeley has uploaded this change for review. (
http://gerrit.cloudera.org:8080/12289
Change subject: KUDU-2662 TestKuduClient.testClientLocation is flaky
......................................................................
KUDU-2662 TestKuduClient.testClientLocation is flaky
At least on Linux, multimaster Kudu tests are run with each master on
its own interface. E.g., in a failed run of testClientLocation, the
three masters were running on 127.26.241.{124,125,126}. Each client is
assigned a location by each master it connects to. The trick was, the
client connection appeared to come from 127.26.241.X for the master on
127.26.241.X. assign-locations.py remembers assigned locations by IP,
so it was attempting to assign 3 locations for one client, plus 3 for
the tablet servers. The test wrongly anticipated 4 assignments and
configured the script to only do 4 assignments, leading to issues like
16:05:46.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:541) W0126
16:05:46.987164 27624 master_service.cc:535] unable to assign location to
client {username='jenkins'}@127.26.241.126:60372: Runtime error: failed to run
location mapping command: Runtime error:
/data/somelongdirectorytoavoidrpathissues/src/kudu/java/kudu-client/build/resources/test/assign-location.py:
process exited with non-zero status 1: Traceback (most recent call last):
16:05:46.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:541) File
"/data/somelongdirectorytoavoidrpathissues/src/kudu/java/kudu-client/build/resources/test/assign-location.py",
line 244, in <module>
16:05:46.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:541)
main()
16:05:46.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:541) File
"/data/somelongdirectorytoavoidrpathissues/src/kudu/java/kudu-client/build/resources/test/assign-location.py",
line 239, in main
16:05:46.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:541)
LocationAssignmentRule(args.location_mapping_rules), args.uid, args.relaxed)
16:05:46.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:541) File
"/data/somelongdirectorytoavoidrpathissues/src/kudu/java/kudu-client/build/resources/test/assign-location.py",
line 198, in get_location
16:05:46.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:541) seq
+ 1, uid, rule.location_mapping_rules, json.dumps(state)))
16:05:46.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:541)
Exception: too many unique identifiers (5) to assign next location to
127.26.241.126 using mapping rules ['/L0:4']. State: {"mapping_rules":
["/L0:4"], "mappings": {"127.26.241.124": "/L0", "127.26.241.65": "/L0",
"127.26.241.66": "/L0", "127.26.241.67": "/L0"}, "seq": 4}
16:05:47.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:541) W0126
16:05:47.073348 27688 master_service.cc:535] unable to assign location to
client {username='jenkins'}@127.26.241.125:41204: Runtime error: failed to run
location mapping command: Runtime error:
/data/somelongdirectorytoavoidrpathissues/src/kudu/java/kudu-client/build/resources/test/assign-location.py:
process exited with non-zero status 1: Traceback (most recent call last):
16:05:47.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:541) File
"/data/somelongdirectorytoavoidrpathissues/src/kudu/java/kudu-client/build/resources/test/assign-location.py",
line 244, in <module>
16:05:47.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:541)
main()
16:05:47.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:541) File
"/data/somelongdirectorytoavoidrpathissues/src/kudu/java/kudu-client/build/resources/test/assign-location.py",
line 239, in main
16:05:47.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:541)
LocationAssignmentRule(args.location_mapping_rules), args.uid, args.relaxed)
16:05:47.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:541) File
"/data/somelongdirectorytoavoidrpathissues/src/kudu/java/kudu-client/build/resources/test/assign-location.py",
line 198, in get_location
16:05:47.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:541) seq
+ 1, uid, rule.location_mapping_rules, json.dumps(state)))
16:05:47.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:541)
Exception: too many unique identifiers (5) to assign next location to
127.26.241.125 using mapping rules ['/L0:4']. State: {"mapping_rules":
["/L0:4"], "mappings": {"127.26.241.124": "/L0", "127.26.241.65": "/L0",
"127.26.241.66": "/L0", "127.26.241.67": "/L0"}, "seq": 4}
This didn't cause a test failure in and of itself- the test would still
pass if the leader master was first to try and assign a location to the
client. So the test would randomly fail.
The solution is just to up the number of allowed assignments. Since our
Java tests are very simple and assign a single location only, this will
avoid the problem.
Change-Id: I2fed3eb62ab9d0f8c1baaa40726355a2a6737b3a
---
M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduClient.java
M java/kudu-test-utils/src/main/java/org/apache/kudu/test/KuduTestHarness.java
2 files changed, 12 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/89/12289/1
--
To view, visit http://gerrit.cloudera.org:8080/12289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I2fed3eb62ab9d0f8c1baaa40726355a2a6737b3a
Gerrit-Change-Number: 12289
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley <[email protected]>
