Impala Public Jenkins has submitted this change and it was merged. Change subject: IMPALA-3794: Workaround for Breakpad ID conflicts ......................................................................
IMPALA-3794: Workaround for Breakpad ID conflicts Breakpad determines the ID of the minidump file to be written in case of a crash during startup of the process randomly, seeded with the current system time with second granularity. If two impalads start up within the same second, there is a chance for a name conflict. The one second delay between starting impalads in start-impala-cluster.py is not sufficient: I0407 22:34:52.018563 28473 minidump.cc:245] Setting minidump size limit to 20971520. I0407 22:34:52.997046 28749 minidump.cc:245] Setting minidump size limit to 20971520. When sending a signal to all of them, one process can overwrite the minidump of another one. This is an upstream issue and is tracked in Breakpad-681. I further confirmed my suspicion by tentatively making an own output folder for each running instance of impalad and was then unable to reproduce the issue. However, it is a more clear solution to fix the underlying issue than to change the folder locations for minidumps in impala. Until this is fixed upstream, we can make sure that we see at least one minidump for the group of impalads in the test cluster. It is not a product defect, since we don't support running multiple impalads on a single host, let alone starting them all at once. To test this I ran the following loop for about an hour on my dev machine without hitting the issue: while [ $? -eq 0 ]; do impala-py.test tests/custom_cluster/test_breakpad.py --exploration_strategy=exhaustive -k test_minidump_relative_path -x -s; done Change-Id: I4ae589f6eb5cbbfb860943214edc0e6415eeb862 Reviewed-on: http://gerrit.cloudera.org:8080/6588 Reviewed-by: Lars Volker <[email protected]> Tested-by: Impala Public Jenkins --- M tests/custom_cluster/test_breakpad.py 1 file changed, 22 insertions(+), 15 deletions(-) Approvals: Impala Public Jenkins: Verified Lars Volker: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/6588 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I4ae589f6eb5cbbfb860943214edc0e6415eeb862 Gerrit-PatchSet: 8 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Lars Volker <[email protected]> Gerrit-Reviewer: Bharath Vissapragada <[email protected]> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker <[email protected]> Gerrit-Reviewer: Michael Brown <[email protected]>
