Re: Help to install Jenkins plugin for flaky tests and review PR

[Benjamin Gonzalez Delgado]

 As a follow-up of this, I created a ticket in INFRA[1] to install the
required plugin, but my ticket couldn't be resolved because I'm not a
committer, I would really appreciate any help on this from a committer.

[1] https://issues.apache.org/jira/browse/INFRA-22672

Benjamin Abraham Gonzalez Delgado | <https://www.wizeline.com/>

Data Engineer

benjamin.gonza...@wizeline.com

Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.


On Wed, Dec 22, 2021 at 6:10 PM Benjamin Gonzalez Delgado <
benjamin.gonza...@wizeline.com> wrote:

> Hello all,
> I am working on BEAM-12672[1], I've already found a way to rerun failed
> tests K times and mark the job status as success, but to report all results
> for each rerun it is required to install a Jenkins plugin [2].
> I've installed that plugin in my local Jenkins environment using the GUI
> and is working as expected.
> Can someone with Jenkins admin permissions and context, help me to install
> the plugin and follow the reviewing of this PR[3]?
> More context of this in the PR comments. Thanks in advance!
>
> [1] https://issues.apache.org/jira/browse/BEAM-12672
> [2] https://plugins.jenkins.io/flaky-test-handler/
> [3] https://github.com/apache/beam/pull/16253
>
> Benjamin Abraham Gonzalez Delgado | <https://www.wizeline.com/>
>
> Data Engineer
>
> benjamin.gonza...@wizeline.com
>
> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
>

-- 
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*

Help to install Jenkins plugin for flaky tests and review PR

[Benjamin Gonzalez Delgado]

Hello all,
I am working on BEAM-12672[1], I've already found a way to rerun failed
tests K times and mark the job status as success, but to report all results
for each rerun it is required to install a Jenkins plugin [2].
I've installed that plugin in my local Jenkins environment using the GUI
and is working as expected.
Can someone with Jenkins admin permissions and context, help me to install
the plugin and follow the reviewing of this PR[3]?
More context of this in the PR comments. Thanks in advance!

[1] https://issues.apache.org/jira/browse/BEAM-12672
[2] https://plugins.jenkins.io/flaky-test-handler/
[3] https://github.com/apache/beam/pull/16253

Benjamin Abraham Gonzalez Delgado | 

Data Engineer

benjamin.gonza...@wizeline.com

Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.

-- 
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*

BEAM-12068: Run Dataflow performance benchmarks on Java 11

[Benjamin Gonzalez Delgado]

Hi Team,
I'm planning to work with BEAM-12068 [1], but I have some related questions.

   - Is it required to create a new Jenkins Job to run the configuration
   with Dataflow V2 and Java11 or do I need to append a new configuration to
   the existing one (e.g., GBK load tests [2])?
   - To execute a test with Dataflow V2, do I just need to pass as args
   
"experiments=beam_fn_api,use_unified_worker,use_runner_v2,shuffle_mode=service"
   or does it require some extra configuration?
   - I followed [3] to configure the metrics local development environment,
   but I had issues with Grafana access credentials, so I recreated with
   docker-compose and custom password in docker-compose.yml [4], but I'm still
   having the same access troubles. Is this the correct setup to work with
   this kind of change? Does someone know how to configure this setup locally?

Any guidance on this would be appreciated.
Thanks!

[1] https://issues.apache.org/jira/browse/BEAM-12068
[2]
https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_LoadTests_GBK_Java.groovy
[3] https://github.com/apache/beam/tree/master/.test-infra/metrics
[4]
https://github.com/apache/beam/blob/master/.test-infra/metrics/docker-compose.yml#L64

-- 
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*

Re: BEAM-11838: An x-lang wrapper for DebeziumIO

[Benjamin Gonzalez Delgado]

Hi Pablo,
It runs perfectly with your changes, I will make some fixes on the
verifiers and changes for minutesToRun, I will upload a PR in a couple of
hours.

Thanks a lot, you really helped me!

>

-- 
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*

Re: BEAM-11838: An x-lang wrapper for DebeziumIO

[Benjamin Gonzalez Delgado]

Hi Pablo!
Did you have a chance to execute the test to find something rare?
Do you know if there is a way to use breakpoints when running this kind of
test I mean breakpoints in java code when running a pipeline from python?


On Fri, May 7, 2021 at 4:54 PM Benjamin Gonzalez Delgado <
benjamin.gonza...@wizeline.com> wrote:

> Hi Pablo!,
>
>
> I run the gradle task
>
> ./gradlew :sdks:python:test-suites:portable:py37:postCommitPy37IT
>
>
> You can comment the other test in
> sdks/python/test-suites/portable/common.gradle
> <https://github.com/benWize/beam/pull/12/files#diff-3b0c84ea390ab209fe959cda93b113ce47f7485f1b6046bbc5597da7d17f784d>
>  for
> a faster run.
>
> Thanks.
>
> On Fri, May 7, 2021 at 4:42 PM Pablo Estrada  wrote:
>
>> Hi Benjamin!
>> How do I run the test on your PR? I can try to run it to see what's going
>> on.[
>> Best
>> -P.
>>
>> On Thu, May 6, 2021 at 1:46 PM Pablo Estrada  wrote:
>>
>>> Hi Benjamin!
>>> That's strange if the Java test is working well that the Python/xlang
>>> test would run into issues. Let me take a look at your PR and see if I can
>>> see anything suspicious.
>>> Best
>>> -P.
>>>
>>> On Thu, May 6, 2021 at 10:49 AM Benjamin Gonzalez Delgado <
>>> benjamin.gonza...@wizeline.com> wrote:
>>>
>>>> Hi team,
>>>> I am working on BEAM-11838[1], but I am a little stuck right now[2], my
>>>> issue is that when I set param maxNumberOfRecords through
>>>> ExternalTransformBuilder(L117) its sets the value correctly to the
>>>> constructor on KafkaSourceConsumerFn (L111), but when it arrives at
>>>> tryClaim method(L307) the value for maxNumberOfRecords is null again and it
>>>> keeps running forever so my test never ends.
>>>> I wonder if some of you have experience with KafkaSourceConsumerFn
>>>> class or creating other cross-language transforms that could help.
>>>> Any guidance on this would be appreciated.
>>>>
>>>> PD. The existing java test DebeziumIOMySqlConnectorIT does not have the
>>>> same problem.
>>>>
>>>> [1] https://issues.apache.org/jira/browse/BEAM-11838
>>>> [2] https://github.com/benWize/beam/pull/12/files#
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *This email and its contents (including any attachments) are being sent
>>>> toyou on the condition of confidentiality and may be protected by
>>>> legalprivilege. Access to this email by anyone other than the intended
>>>> recipientis unauthorized. If you are not the intended recipient, please
>>>> immediatelynotify the sender by replying to this message and delete the
>>>> materialimmediately from your system. Any further use, dissemination,
>>>> distributionor reproduction of this email is strictly prohibited. Further,
>>>> norepresentation is made with respect to any content contained in this
>>>> email.*
>>>
>>>

-- 
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*

Re: BEAM-11838: An x-lang wrapper for DebeziumIO

[Benjamin Gonzalez Delgado]

Hi Pablo!,


I run the gradle task

./gradlew :sdks:python:test-suites:portable:py37:postCommitPy37IT


You can comment the other test in
sdks/python/test-suites/portable/common.gradle
<https://github.com/benWize/beam/pull/12/files#diff-3b0c84ea390ab209fe959cda93b113ce47f7485f1b6046bbc5597da7d17f784d>
for
a faster run.

Thanks.

On Fri, May 7, 2021 at 4:42 PM Pablo Estrada  wrote:

> Hi Benjamin!
> How do I run the test on your PR? I can try to run it to see what's going
> on.[
> Best
> -P.
>
> On Thu, May 6, 2021 at 1:46 PM Pablo Estrada  wrote:
>
>> Hi Benjamin!
>> That's strange if the Java test is working well that the Python/xlang
>> test would run into issues. Let me take a look at your PR and see if I can
>> see anything suspicious.
>> Best
>> -P.
>>
>> On Thu, May 6, 2021 at 10:49 AM Benjamin Gonzalez Delgado <
>> benjamin.gonza...@wizeline.com> wrote:
>>
>>> Hi team,
>>> I am working on BEAM-11838[1], but I am a little stuck right now[2], my
>>> issue is that when I set param maxNumberOfRecords through
>>> ExternalTransformBuilder(L117) its sets the value correctly to the
>>> constructor on KafkaSourceConsumerFn (L111), but when it arrives at
>>> tryClaim method(L307) the value for maxNumberOfRecords is null again and it
>>> keeps running forever so my test never ends.
>>> I wonder if some of you have experience with KafkaSourceConsumerFn class
>>> or creating other cross-language transforms that could help.
>>> Any guidance on this would be appreciated.
>>>
>>> PD. The existing java test DebeziumIOMySqlConnectorIT does not have the
>>> same problem.
>>>
>>> [1] https://issues.apache.org/jira/browse/BEAM-11838
>>> [2] https://github.com/benWize/beam/pull/12/files#
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *This email and its contents (including any attachments) are being sent
>>> toyou on the condition of confidentiality and may be protected by
>>> legalprivilege. Access to this email by anyone other than the intended
>>> recipientis unauthorized. If you are not the intended recipient, please
>>> immediatelynotify the sender by replying to this message and delete the
>>> materialimmediately from your system. Any further use, dissemination,
>>> distributionor reproduction of this email is strictly prohibited. Further,
>>> norepresentation is made with respect to any content contained in this
>>> email.*
>>
>>

-- 
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*

BEAM-11838: An x-lang wrapper for DebeziumIO

[Benjamin Gonzalez Delgado]

Hi team,
I am working on BEAM-11838[1], but I am a little stuck right now[2], my
issue is that when I set param maxNumberOfRecords through
ExternalTransformBuilder(L117) its sets the value correctly to the
constructor on KafkaSourceConsumerFn (L111), but when it arrives at
tryClaim method(L307) the value for maxNumberOfRecords is null again and it
keeps running forever so my test never ends.
I wonder if some of you have experience with KafkaSourceConsumerFn class or
creating other cross-language transforms that could help.
Any guidance on this would be appreciated.

PD. The existing java test DebeziumIOMySqlConnectorIT does not have the
same problem.

[1] https://issues.apache.org/jira/browse/BEAM-11838
[2] https://github.com/benWize/beam/pull/12/files#

-- 
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*

BEAM-3713: Moving from nose to pytest

[Benjamin Gonzalez Delgado]

Hi team,
I am planning to work in BEAM-3713
, but I see there are PRs
related to the task.
Could someone guide me on the work that remains missing regarding the
migration from nose to pytest?
Any guidance on this would be appreciated.

Thanks!
Benjamin

-- 
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*

Re: Add benglez to JIRA contributor list

[Benjamin Gonzalez Delgado]

Thanks Pablo!

On Tue, Mar 9, 2021 at 11:35 AM Pablo Estrada  wrote:

> Hi Benjamin!
> Welcome to Beam. I've added you as a contributor, and now you can assign
> issues to yourself on JIRA.
> Thanks,
> -P.
>
> On Tue, Mar 9, 2021 at 8:52 AM Benjamin Gonzalez Delgado <
> benjamin.gonza...@wizeline.com> wrote:
>
>> This is Benjamin from Wizeline. I will be working on some issues. Can
>> someone add me as a contributor to Beam's Jira issue tracker? I would like
>> to create/assign tickets for my work.
>> My Jira ID is benglez
>>
>> Thanks in advance!
>>
>>
>>
>>
>>
>>
>>
>>
>> *This email and its contents (including any attachments) are being sent
>> toyou on the condition of confidentiality and may be protected by
>> legalprivilege. Access to this email by anyone other than the intended
>> recipientis unauthorized. If you are not the intended recipient, please
>> immediatelynotify the sender by replying to this message and delete the
>> materialimmediately from your system. Any further use, dissemination,
>> distributionor reproduction of this email is strictly prohibited. Further,
>> norepresentation is made with respect to any content contained in this
>> email.*
>
>

-- 
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*

Add benglez to JIRA contributor list

[Benjamin Gonzalez Delgado]

This is Benjamin from Wizeline. I will be working on some issues. Can
someone add me as a contributor to Beam's Jira issue tracker? I would like
to create/assign tickets for my work.
My Jira ID is benglez

Thanks in advance!

-- 
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*