Re: Anyone have any ideas why beam_PreCommit_CommunityMetrics is failing?

[Daniela Martín]

Thanks for the review Cham.

Regards,

On Wed, Apr 20, 2022 at 8:28 PM Chamikara Jayalath 
wrote:

> Thanks for the fix. Merged the PR.
>
> - Cham
>
> On Wed, Apr 20, 2022 at 1:37 PM Daniela Martín <
> daniela.mar...@wizeline.com> wrote:
>
>> Hi everyone,
>>
>> After a deep testing along with @Elias Segundo Antonio
>>   (he is currently working on BEAM-14169
>> ), we found that the issue is
>> in fact related to the rotation of the k8s credentials. We identified that
>> in this specific test, there is no get-credentials instruction when it’s
>> executed, that’s why the test is failing.
>>
>> In order to avoid this issue, we have refactored the test including the
>> get-credentials and remove-config every time the job is triggered.  Also,
>> we have updated the --dry-run flag in order to remove the warning of the
>> deprecated usage.
>>
>> Could you please help us to review the PR#17396
>> ?
>>
>> Please let us know if you have any comments or questions.
>>
>> Thank you!
>>
>> Regards,
>>
>> On Wed, Apr 13, 2022 at 12:03 PM Daniela Martín <
>> daniela.mar...@wizeline.com> wrote:
>>
>>> Hi everyone,
>>>
>>> I'll take a look. Thank you for the information.
>>>
>>> Regards,
>>>
>>> On Mon, Mar 14, 2022 at 5:11 PM Ahmet Altay  wrote:
>>>
 I do not know the code well enough either. But I could not find any
 references to "104.154.102.21" in the code search.

 In case this might help with anyone to help here:
 - Failing test is a single line of kubectl command "kubectl apply
 --dry-run=true -Rf kubernetes" (
 https://github.com/apache/beam/blob/f779a3fca31f08ada5011155484b69bdca962754/.test-infra/metrics/build.gradle#L55
 )
 - kubernetes is referring to the directory with yaml files (
 https://github.com/apache/beam/tree/master/.test-infra/metrics/kubernetes
 )

 On Mon, Mar 14, 2022 at 7:40 AM Kerry Donny-Clark 
 wrote:

> Hi Daniel,
> I may be the culprit, as I had to rotate our credentials on tke k8
> cluster. That meant I also had to rebuild the nodes, and perform an IP
> rotation. My suspicion is that there may be hardcoded addresses that
> changed when the nodes were rebuilt, but I don't know the code well enough
> to find out if that's true.
>
> Kerry
>
> On Thu, Mar 10, 2022 at 6:11 PM Daniel Oliveira <
> danolive...@google.com> wrote:
>
>> Hi everyone,
>>
>> Can anyone take some time to look at BEAM-14017
>> ? Especially if
>> you're at all familiar with our scripts for gathering metrics for the
>> Community Metrics page.
>>
>> I noticed beam_PreCommit_CommunityMetrics_Cron is failing
>> consistently and I took a look into it (everything is documented in the
>> Jira). But I found very little to help diagnose the problem. As best I 
>> can
>> tell, the test is failing to connect to some Kubernetes cluster, but for
>> some reason the Community Metrics Page
>>  it's supposed
>> to update is still getting regularly updated. I don't really have the 
>> time
>> to look into it further so I'm hoping someone else can take a look.
>>
>> Thanks,
>> Daniel Oliveira
>>
>
>>>
>>> --
>>>
>>> Daniela Martín (She/Her) | 
>>>
>>> Site Reliability Engineer III
>>>
>>> daniela.mar...@wizeline.com
>>>
>>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
>>>
>>> Follow us Twitter  | Facebook
>>>  | Instagram
>>>  | LinkedIn
>>> 
>>>
>>> Share feedback on Clutch 
>>>
>>
>>
>> --
>>
>> Daniela Martín (She/Her) | 
>>
>> Site Reliability Engineer III
>>
>> daniela.mar...@wizeline.com
>>
>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
>>
>> Follow us Twitter  | Facebook
>>  | Instagram
>>  | LinkedIn
>> 
>>
>> Share feedback on Clutch 
>>
>>
>>
>>
>>
>>
>>
>>
>> *This email and its contents (including any attachments) are being sent
>> toyou on the condition of confidentiality and may be protected by
>> legalprivilege. Access to this email by anyone other than the intended
>> recipientis unauthorized. If you are not the intended recipient, please
>> immediatelynotify the sender by replying to this message and delete the
>> materialimmediately from your system. Any further use, dissemination,
>> distributionor reproduction of this email is strictly 

Re: Anyone have any ideas why beam_PreCommit_CommunityMetrics is failing?

[Daniela Martín]

Hi everyone,

After a deep testing along with @Elias Segundo Antonio
  (he is currently working on BEAM-14169
), we found that the issue is in
fact related to the rotation of the k8s credentials. We identified that in
this specific test, there is no get-credentials instruction when it’s
executed, that’s why the test is failing.

In order to avoid this issue, we have refactored the test including the
get-credentials and remove-config every time the job is triggered.  Also,
we have updated the --dry-run flag in order to remove the warning of the
deprecated usage.

Could you please help us to review the PR#17396
?

Please let us know if you have any comments or questions.

Thank you!

Regards,

On Wed, Apr 13, 2022 at 12:03 PM Daniela Martín 
wrote:

> Hi everyone,
>
> I'll take a look. Thank you for the information.
>
> Regards,
>
> On Mon, Mar 14, 2022 at 5:11 PM Ahmet Altay  wrote:
>
>> I do not know the code well enough either. But I could not find any
>> references to "104.154.102.21" in the code search.
>>
>> In case this might help with anyone to help here:
>> - Failing test is a single line of kubectl command "kubectl apply
>> --dry-run=true -Rf kubernetes" (
>> https://github.com/apache/beam/blob/f779a3fca31f08ada5011155484b69bdca962754/.test-infra/metrics/build.gradle#L55
>> )
>> - kubernetes is referring to the directory with yaml files (
>> https://github.com/apache/beam/tree/master/.test-infra/metrics/kubernetes
>> )
>>
>> On Mon, Mar 14, 2022 at 7:40 AM Kerry Donny-Clark 
>> wrote:
>>
>>> Hi Daniel,
>>> I may be the culprit, as I had to rotate our credentials on tke k8
>>> cluster. That meant I also had to rebuild the nodes, and perform an IP
>>> rotation. My suspicion is that there may be hardcoded addresses that
>>> changed when the nodes were rebuilt, but I don't know the code well enough
>>> to find out if that's true.
>>>
>>> Kerry
>>>
>>> On Thu, Mar 10, 2022 at 6:11 PM Daniel Oliveira 
>>> wrote:
>>>
 Hi everyone,

 Can anyone take some time to look at BEAM-14017
 ? Especially if
 you're at all familiar with our scripts for gathering metrics for the
 Community Metrics page.

 I noticed beam_PreCommit_CommunityMetrics_Cron is failing consistently
 and I took a look into it (everything is documented in the Jira). But I
 found very little to help diagnose the problem. As best I can tell, the
 test is failing to connect to some Kubernetes cluster, but for some reason
 the Community Metrics Page
  it's supposed to
 update is still getting regularly updated. I don't really have the time to
 look into it further so I'm hoping someone else can take a look.

 Thanks,
 Daniel Oliveira

>>>
>
> --
>
> Daniela Martín (She/Her) | 
>
> Site Reliability Engineer III
>
> daniela.mar...@wizeline.com
>
> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
>
> Follow us Twitter  | Facebook
>  | Instagram
>  | LinkedIn
> 
>
> Share feedback on Clutch 
>


-- 

Daniela Martín (She/Her) | 

Site Reliability Engineer III

daniela.mar...@wizeline.com

Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.

Follow us Twitter  | Facebook
 | Instagram
 | LinkedIn


Share feedback on Clutch 

-- 
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*

Re: Anyone have any ideas why beam_PreCommit_CommunityMetrics is failing?

[Daniela Martín]

Hi everyone,

I'll take a look. Thank you for the information.

Regards,

On Mon, Mar 14, 2022 at 5:11 PM Ahmet Altay  wrote:

> I do not know the code well enough either. But I could not find any
> references to "104.154.102.21" in the code search.
>
> In case this might help with anyone to help here:
> - Failing test is a single line of kubectl command "kubectl apply
> --dry-run=true -Rf kubernetes" (
> https://github.com/apache/beam/blob/f779a3fca31f08ada5011155484b69bdca962754/.test-infra/metrics/build.gradle#L55
> )
> - kubernetes is referring to the directory with yaml files (
> https://github.com/apache/beam/tree/master/.test-infra/metrics/kubernetes)
>
> On Mon, Mar 14, 2022 at 7:40 AM Kerry Donny-Clark 
> wrote:
>
>> Hi Daniel,
>> I may be the culprit, as I had to rotate our credentials on tke k8
>> cluster. That meant I also had to rebuild the nodes, and perform an IP
>> rotation. My suspicion is that there may be hardcoded addresses that
>> changed when the nodes were rebuilt, but I don't know the code well enough
>> to find out if that's true.
>>
>> Kerry
>>
>> On Thu, Mar 10, 2022 at 6:11 PM Daniel Oliveira 
>> wrote:
>>
>>> Hi everyone,
>>>
>>> Can anyone take some time to look at BEAM-14017
>>> ? Especially if
>>> you're at all familiar with our scripts for gathering metrics for the
>>> Community Metrics page.
>>>
>>> I noticed beam_PreCommit_CommunityMetrics_Cron is failing consistently
>>> and I took a look into it (everything is documented in the Jira). But I
>>> found very little to help diagnose the problem. As best I can tell, the
>>> test is failing to connect to some Kubernetes cluster, but for some reason
>>> the Community Metrics Page
>>>  it's supposed to
>>> update is still getting regularly updated. I don't really have the time to
>>> look into it further so I'm hoping someone else can take a look.
>>>
>>> Thanks,
>>> Daniel Oliveira
>>>
>>

-- 

Daniela Martín (She/Her) | 

Site Reliability Engineer III

daniela.mar...@wizeline.com

Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.

Follow us Twitter  | Facebook
 | Instagram
 | LinkedIn


Share feedback on Clutch 

-- 
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*