[
https://issues.apache.org/jira/browse/BEAM-4041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16432075#comment-16432075
]
Łukasz Gajowy edited comment on BEAM-4041 at 4/10/18 3:13 PM:
--------------------------------------------------------------
[~alanmyrvold], [~chamikara] do you think that this could be related to some
Google Cloud IP addresses quotas?
When the Performance tests are run on jenkins, there are 7 of them trying to
obtain LoadBalancer address in different namespaces but at the same time. Maybe
kubernetes simply has no address left that could be assigned to the
LoadBalancer?
Currently, the tests that use kubernetes and are started simultaneously are:
- MongoDBIOIT
- HadoopInputFormatIOIT
- JdbcIOIT
- file based tests that use HDFS (added in this PR:
[https://github.com/apache/beam/pull/4861], 4 of them)
Alternatively, we could try to increase the waiting for LoadBalancerIP time in
Perfkit, but I'm not sure it's a real issue solver.
was (Author: łukaszg):
[~alanmyrvold], [~chamikara] do you think that this could be related to some
Google Cloud IP addresses quotas?
When the Performance tests are run on jenkins, there are 7 of them trying to
obtain LoadBalancer address in different namespaces but at the same time. Maybe
kubernetes simply has no address left that could be assigned to the
LoadBalancer?
Currently, the tests that use kubernetes and are started simultaneously are:
- MongoDBIOIT
- HadoopInputFormatIOIT
- JdbcIOIT
- file based tests that use HDFS (added in this PR:
[https://github.com/apache/beam/pull/4861], 4 of them)
Alternatively, we could try to increase the waiting for LoadBalancerIP time in
Perfkit, but I'm not sure it's a real issue solver.
> Performance tests fail due to kubernetes load balancer problems
> ---------------------------------------------------------------
>
> Key: BEAM-4041
> URL: https://issues.apache.org/jira/browse/BEAM-4041
> Project: Beam
> Issue Type: Bug
> Components: testing
> Reporter: Łukasz Gajowy
> Assignee: Jason Kuster
> Priority: Major
>
> Recently, as we added more IOITs to be run on jenkins using kubernetes, some
> of them started to fail randomly, because they couldn't retrieve LoadBalancer
> address. Normally obtaining the address took about one minute. Perfkit waits
> for the address (actively checking for it) for 3 minutes. This should be
> enough for getting the address, yet it recently started to exceed the 3
> minutes limit. I also noticed that this error didn't happen when there were
> fewer tests.
> Example logs:
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_Compressed_TextIOIT_HDFS/31/console
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
