Kamil Szewczyk commented on BEAM-4041:

[~alanmyrvold] [~chamikara] [~jasonkuster] [~ŁukaszG]

I have also took a look at that issue and noticed that  we spin our 
prerformance tests each six hours, so that many jobs try to run at the same 
time. It's possible to easily run them each six hours, but with little time 
shift ( like +15 min), by modifying cron job settings. This could probably 
help, but I couldn't believe that providing just a few external IPs more at 
single point of time is an issue for GCE. It's just my guess that maybe it 
could help.

> Performance tests fail due to kubernetes load balancer problems
> ---------------------------------------------------------------
>                 Key: BEAM-4041
>                 URL: https://issues.apache.org/jira/browse/BEAM-4041
>             Project: Beam
>          Issue Type: Bug
>          Components: testing
>            Reporter: Łukasz Gajowy
>            Assignee: Jason Kuster
>            Priority: Major
> Recently, as we added more IOITs to be run on jenkins using kubernetes, some 
> of them started to fail randomly, because they couldn't retrieve LoadBalancer 
> address. Normally obtaining the address took about one minute. Perfkit waits 
> for the address (actively checking for it) for 3 minutes. This should be 
> enough for getting the address, yet it recently started to exceed the 3 
> minutes limit. I also noticed that this error didn't happen when there were 
> fewer tests.
> Example logs:
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_Compressed_TextIOIT_HDFS/31/console

This message was sent by Atlassian JIRA

Reply via email to