Re: Contributor permission for Beam Jira tickets

[Kenneth Knowles]

Done! Welcome!

On Wed, Jan 12, 2022 at 2:06 PM Andoni Guzman Becerra <
andoni.guz...@wizeline.com> wrote:

> Hi all!
> I'm Andoni Guzman from Wizeline Team.
> I'll be working with the wizeline team in the apache beam project. Can
> someone add me as a contributor for Beam's Jira issue tracker? I would like
> to create/assign tickets for my work.
> My user in jira is:  andoni.guzman
>
> Thanks,
> Andoni Guzman
>
>
>
>
>
>
>
>
>
> *This email and its contents (including any attachments) are being sent
> toyou on the condition of confidentiality and may be protected by
> legalprivilege. Access to this email by anyone other than the intended
> recipientis unauthorized. If you are not the intended recipient, please
> immediatelynotify the sender by replying to this message and delete the
> materialimmediately from your system. Any further use, dissemination,
> distributionor reproduction of this email is strictly prohibited. Further,
> norepresentation is made with respect to any content contained in this
> email.*

Re: Jira contributor permissions

[Kenneth Knowles]

Done! Welcome

On Wed, Jan 12, 2022 at 1:40 PM Ahmed Abualsaud 
wrote:

> Hey devs team,
>
> I'm Ahmed, joining from Google's GCP Dataflow team (focused on Beam IO).
> Can someone add me as a contributor for Beam's issue tracker? I'd like to
> assign myself to BEAM-10584 to get a feel of things. My Jira ID is
> *ahmedabu*.
>
> Looking forward to working with you all,
> Ahmed
>

Re: Question about Go SDK Built-in I/O Transforms

[Robert Burke]

Hello Leonardo!

I'm happy to hear of your interest in the Go SDK! The SDK is recently out
of experimental, but is not yet officially supported by Dataflow. (It
works, and we test the SDK on Dataflow, but user support is at the
discretion of the Dataflow side at this time.)

The short answer is yes, these transforms can be available in future
releases.

The longer answer is the following:

There are still some gaps between the Go SDK and Java and Python SDKs. For
some of these we use Cross Language Transforms, which lets pipelines insert
transforms from other SDKs into their Go pipelines.
For example, this allows Go pipelines to make use of the Java KafkaIO
transform. See
https://beam.apache.org/documentation/programming-guide/#1323-using-cross-language-transforms-in-a-go-pipeline
which will be kept up to date with the latest state.

As described at that link, automatic startup of the expansion service isn't
available yet, but it's almost there. The overall use is being worked on
presently, and should start to become available by v2.37.0 .

The same mechanism will be used to add BigTable support. I don't know about
Elastic, but if Java has it, Go and Python can wrap it.

If you're keen on contributing a solution for yourself, if you follow the
example set by Kafka, we would welcome contributions of those wrappers for
the Go SDK too.

Other than cross language, there's potentially the option to write a native
Go transform. However, the Go SDK doesn't support native unbounded source
transforms. It requires a feature called DoFn Self Checkpointing, that's
not yet implemented in the SDK. It is planned though. This doesn't prevent
streaming IOs from cross language being used though.

 Scalable Bounded transforms can be written using Splittable DoFns however.
https://beam.apache.org/documentation/programming-guide/#splittable-dofns

Every version, the SDK gets closer to being fully featured with the Beam
Model. It's exciting!

I'd love to hear more about your use case, so we can see if the Go SDK can
get you there.

Robert Burke
Beam Go Busybody


On Wed, Jan 12, 2022, 1:11 PM Leonardo Reis 
wrote:

> Hello everyone, how are you?
>
> My name is Leonardo and I'm from Brazil. In my current project, we are
> thinking of implementing Apache Beam with Go SDK to run our jobs with
> Dataflow runner. But in our architecture we have some external dependencies
> like Kafka Streams, Bigtable and Elastic and we didn't find any
> transformation for them.
>
> Will these IO transformations exist in future releases?
>
> Do you have any suggestions on how we can handle these dependencies using
> the Go SDK?
>
> Best regards,
> Leonardo Reis
>
> Data Engineer
> (16) 3509-
> leonardo.r...@arquivei.com.br 
> arquivei.com.br
> 
>  [image: facebook]  [image:
> linkedin]  [image: instagram]
> 
>

Re: [DISCUSS] Migrate Jira to GitHub Issues?

[Aizhamal Nurmamat kyzy]

Hi all,

Is there a consensus to migrate to GitHub?

On Wed, Dec 15, 2021 at 9:17 AM Brian Hulette  wrote:

>
>
> On Tue, Dec 14, 2021 at 1:14 PM Kenneth Knowles  wrote:
>
>>
>>
>> On Thu, Dec 9, 2021 at 11:50 PM Jean-Baptiste Onofre 
>> wrote:
>>
>>> Hi,
>>>
>>> No problem for me. The only thing I don’t like with GitHub issues is
>>> that fact that it’s not possible to “assign” several milestones to an issue.
>>> When we maintain several active branch/version, it sucks (one issue ==
>>> one milestone), as we have to create several issue.
>>>
>>
>> This is a good point to consider. In Beam we often create multiple issues
>> anyhow when we intend to backport/cherrypick a fix. One issue for the
>> original fix and one each targeted cherrypick. This way their resolution
>> status can be tracked separately. But it is nice for users to be able to go
>> back and edit the original bug report to say which versions are affected
>> and which are not.
>>
>
> I looked into this a little bit. It looks like milestones don't have to
> represent a release (e.g. they could represent some abstract goal), but
> they are often associated with releases. This seems like a reasonable field
> to map to "Fix Version/s" in jira, but jira does support specifying
> multiple releases. So one issue == one milestone would be a regression.
> As Kenn pointed out though we often create a separate jira to track
> backports anyway (even though we could just specify multiple fix versions),
> so I'm not sure this is a significant blocker.
>
> If we want to use milestones to track abstract goals, I think we'd be out
> of luck. We could just use labels, but the GitHub UI doesn't present a nice
> burndown chart for those. See
> https://github.com/pandas-dev/pandas/milestones vs.
> https://github.com/pandas-dev/pandas/labels. FWIW jira doesn't have great
> functionality here either.
>
>
>>
>> Kenn
>>
>>
>>>
>>> Regards
>>> JB
>>>
>>> > Le 10 déc. 2021 à 01:28, Kyle Weaver  a écrit :
>>> >
>>> > I’m in favor of switching to Github issues. I can’t think of a single
>>> thing jira does better.
>>> >
>>> > Thanks Jarek, this is a really great resource [1]. For another
>>> reference, the Calcite project is engaged in the same discussion right now
>>> [2]. I came up with many of the same points independently before I saw
>>> their thread.
>>> >
>>> > When evaluating feature parity, we should make a distinction between
>>> non-structured (text) and structured data. And we don’t need a strict
>>> mechanical mapping for everything unless we’re planning on automatically
>>> migrating all existing issues. I don’t see the point in automatic
>>> migration, though; as Jarek pointed out, we’d end up perpetuating a ton of
>>> obsolete issues.
>>> >
>>> >   • We use nested issues and issue relations in jira, but as far
>>> as I know robots don’t use them and we don’t query them much, so we’re not
>>> losing anything by moving from an API to plain English descriptions: “This
>>> issue is blocked by issue #n.” Mentions show up automatically on other
>>> issues.
>>> >   • For component, type, priority, etc., we can use Github labels.
>>> >   • Version(s) affected is used inconsistently, and as far as I
>>> know only by humans, so a simple English description is fine. We can follow
>>> the example of other projects and make the version affected a part of the
>>> issue template.
>>> >   • For fix version, which we use to track which issues we want to
>>> fix in upcoming releases, as well as automatically generate release notes:
>>> Github has “milestones,” which can be marked on PRs or issues, or both.
>>> >   • IMO the automatically generated JIRA release notes are
>>> not especially useful anyway. They are too detailed for a quick summary,
>>> and not precise enough to show everything. For a readable summary, we use
>>> CHANGES.md to highlight changes we especially want users to know about. For
>>> a complete list of changes, there’s the git commit log, which is the
>>> ultimate source of truth.
>>> >   • We’d only want to preserve reporter and assignee if we’re
>>> planning on migrating everything automatically, and even then I think it’d
>>> be fine to compile a map of active contributors and drop the rest.
>>> >
>>> > As for the advantages of switching (just the ones off the top of my
>>> head):
>>> >   • As others have mentioned, it’s less burden for new
>>> contributors to create new issues and comment on existing ones.
>>> >   • Effortless linking between issues and PRs.
>>> >   • Github -> jira links were working for a short while,
>>> but they seem to be broken at the moment.
>>> >   • Jira -> github links only show: “links to GitHub Pull
>>> Request #x”. They don’t say the status of the PR, so you have to follow
>>> the link to find out. Especially inconvenient when one jira maps to several
>>> PRs, and you have to open all the links to get a summary of what work was
>>> done.
>>> > 

Contributor permission for Beam Jira tickets

[Andoni Guzman Becerra]

Hi all!
I'm Andoni Guzman from Wizeline Team.
I'll be working with the wizeline team in the apache beam project. Can
someone add me as a contributor for Beam's Jira issue tracker? I would like
to create/assign tickets for my work.
My user in jira is:  andoni.guzman

Thanks,
Andoni Guzman

-- 
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*

Question about Go SDK Built-in I/O Transforms

[Leonardo Reis]

Hello everyone, how are you?

My name is Leonardo and I'm from Brazil. In my current project, we are
thinking of implementing Apache Beam with Go SDK to run our jobs with
Dataflow runner. But in our architecture we have some external dependencies
like Kafka Streams, Bigtable and Elastic and we didn't find any
transformation for them.

Will these IO transformations exist in future releases?

Do you have any suggestions on how we can handle these dependencies using
the Go SDK?

Best regards,
Leonardo Reis

Data Engineer
(16) 3509-
leonardo.r...@arquivei.com.br 
arquivei.com.br

 [image: facebook]  [image:
linkedin]  [image: instagram]

SDF/Unbounded/Stream tests examples

[Marco Robles]

Hi,

I am looking for *integration tests for SDF implementation* (testing the
pipeline) for unbounded-stream that may help me as an *example* to validate
some expected behaviors in my current implementation for the SDF tests for
PulsarIO connector.

I was using KafkaIO as an example for the testing, but it seems the SDF
version for the testing is not being used (Pipeline tests

).

Thanks in advance

-- 

*Marco Robles* *|* WIZELINE

Software Engineer

marco.rob...@wizeline.com

-- 
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*

Re: Developing on an M1 Mac

[Jarek Potiuk]

Comment from the side - If you use Docker - experience from Airflow -
until we will get ARM images, docker experience is next to unusable
(docker filesystem slowness + emulation).

J.

On Wed, Jan 12, 2022 at 6:21 PM Daniel Collins  wrote:
>
> I regularly develop on a non-m1 mac using intellij, which mostly works out of 
> the box. Are you running into any particular issues building or just looking 
> for advice?
>
> -Daniel
>
> On Wed, Jan 12, 2022 at 12:16 PM Matt Rudary  wrote:
>>
>> Does anyone do Beam development on an M1 Mac? Any tips to getting things up 
>> and running?
>>
>>
>>
>> Alternatively, does anyone have a good “workstation in the cloud” setup?
>>
>>
>>
>> Thanks
>>
>> Matt

Developing on an M1 Mac

[Matt Rudary]

Does anyone do Beam development on an M1 Mac? Any tips to getting things up and 
running?

Alternatively, does anyone have a good "workstation in the cloud" setup?

Thanks
Matt

Flaky test issue report (41)

[Beam Jira Bot]

This is your daily summary of Beam's current flaky tests 
(https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20labels%20%3D%20flake)

These are P1 issues because they have a major negative impact on the community 
and make it hard to determine the quality of the software.

https://issues.apache.org/jira/browse/BEAM-13611: 
CrossLanguageJdbcIOTest.test_xlang_jdbc_write failing in Python PostCommits 
(created 2022-01-07)
https://issues.apache.org/jira/browse/BEAM-13575: Flink 
testParDoRequiresStableInput flaky (created 2021-12-28)
https://issues.apache.org/jira/browse/BEAM-13525: Java VR (Dataflow, V2, 
Streaming) failing: ParDoTest$TimestampTests/OnWindowExpirationTests (created 
2021-12-22)
https://issues.apache.org/jira/browse/BEAM-13519: Java precommit flaky 
(timing out) (created 2021-12-22)
https://issues.apache.org/jira/browse/BEAM-13453: Flake in 
org.apache.beam.sdk.io.mqtt.MqttIOTest.testReadObject: Address already in use 
(created 2021-12-13)
https://issues.apache.org/jira/browse/BEAM-13401: 
beam_PostCommit_Java_DataflowV2 
org.apache.beam.sdk.io.gcp.pubsublite.ReadWriteIT flaky (created 2021-12-07)
https://issues.apache.org/jira/browse/BEAM-13393: GroupIntoBatchesTest is 
failing (created 2021-12-07)
https://issues.apache.org/jira/browse/BEAM-13367: 
[beam_PostCommit_Python36] [ 
apache_beam.io.gcp.experimental.spannerio_read_it_test] Failure summary 
(created 2021-12-01)
https://issues.apache.org/jira/browse/BEAM-13312: 
org.apache.beam.sdk.transforms.ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundle
 is flaky in Java Spark ValidatesRunner suite  (created 2021-11-23)
https://issues.apache.org/jira/browse/BEAM-13311: 
org.apache.beam.sdk.transforms.ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElementStateful
 is flaky in Java ValidatesRunner Flink suite. (created 2021-11-23)
https://issues.apache.org/jira/browse/BEAM-13234: Flake in 
StreamingWordCountIT.test_streaming_wordcount_it (created 2021-11-12)
https://issues.apache.org/jira/browse/BEAM-13025: pubsublite.ReadWriteIT 
flaky in beam_PostCommit_Java_DataflowV2   (created 2021-10-08)
https://issues.apache.org/jira/browse/BEAM-12928: beam_PostCommit_Python36 
- CrossLanguageSpannerIOTest - flakey failing (created 2021-09-21)
https://issues.apache.org/jira/browse/BEAM-12859: 
org.apache.beam.runners.dataflow.worker.fn.logging.BeamFnLoggingServiceTest.testMultipleClientsFailingIsHandledGracefullyByServer
 is flaky (created 2021-09-08)
https://issues.apache.org/jira/browse/BEAM-12858: 
org.apache.beam.sdk.io.gcp.datastore.RampupThrottlingFnTest.testRampupThrottler 
is flaky (created 2021-09-08)
https://issues.apache.org/jira/browse/BEAM-12809: 
testTwoTimersSettingEachOtherWithCreateAsInputBounded flaky (created 2021-08-26)
https://issues.apache.org/jira/browse/BEAM-12794: 
PortableRunnerTestWithExternalEnv.test_pardo_timers flaky (created 2021-08-24)
https://issues.apache.org/jira/browse/BEAM-12793: 
beam_PostRelease_NightlySnapshot failed (created 2021-08-24)
https://issues.apache.org/jira/browse/BEAM-12766: Already Exists: Dataset 
apache-beam-testing:python_bq_file_loads_NNN (created 2021-08-16)
https://issues.apache.org/jira/browse/BEAM-12515: Python PreCommit flaking 
in PipelineOptionsTest.test_display_data (created 2021-06-18)
https://issues.apache.org/jira/browse/BEAM-12322: Python precommit flaky: 
Failed to read inputs in the data plane (created 2021-05-10)
https://issues.apache.org/jira/browse/BEAM-12320: 
PubsubTableProviderIT.testSQLSelectsArrayAttributes[0] failing in SQL 
PostCommit (created 2021-05-10)
https://issues.apache.org/jira/browse/BEAM-12291: 
org.apache.beam.runners.flink.ReadSourcePortableTest.testExecution[streaming: 
false] is flaky (created 2021-05-05)
https://issues.apache.org/jira/browse/BEAM-12200: 
SamzaStoreStateInternalsTest is flaky (created 2021-04-20)
https://issues.apache.org/jira/browse/BEAM-12163: Python GHA PreCommits 
flake with grpc.FutureTimeoutError on SDK harness startup (created 2021-04-13)
https://issues.apache.org/jira/browse/BEAM-12061: beam_PostCommit_SQL 
failing on KafkaTableProviderIT.testFakeNested (created 2021-03-27)
https://issues.apache.org/jira/browse/BEAM-11837: Java build flakes: 
"Memory constraints are impeding performance" (created 2021-02-18)
https://issues.apache.org/jira/browse/BEAM-11661: hdfsIntegrationTest 
flake: network not found (py38 postcommit) (created 2021-01-19)
https://issues.apache.org/jira/browse/BEAM-11641: Bigquery Read tests are 
flaky on Flink runner in Python PostCommit suites (created 2021-01-15)
https://issues.apache.org/jira/browse/BEAM-11541: 
testTeardownCalledAfterExceptionInProcessElement flakes on direct runner. 
(created 2020-12-30)
https://issues.apache.org/jira/browse/BEAM-10955: Flink Java Runner test 
flake: Could not find Flink job 

P1 issues report (62)

[Beam Jira Bot]

This is your daily summary of Beam's current P1 issues, not including flaky 
tests 
(https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20priority%20%3D%20P1%20AND%20(labels%20is%20EMPTY%20OR%20labels%20!%3D%20flake).

See https://beam.apache.org/contribute/jira-priorities/#p1-critical for the 
meaning and expectations around P1 issues.

https://issues.apache.org/jira/browse/BEAM-13616: Update protobuf-java to 
3.19.2 and other vendored dependencies that use protobuf (created 2022-01-08)
https://issues.apache.org/jira/browse/BEAM-13615: Bumping up FnApi 
environment version to 9 in Java, Python SDK (created 2022-01-07)
https://issues.apache.org/jira/browse/BEAM-13611: 
CrossLanguageJdbcIOTest.test_xlang_jdbc_write failing in Python PostCommits 
(created 2022-01-07)
https://issues.apache.org/jira/browse/BEAM-13606: bigtable io doesn't 
handle non-ok row mutations (created 2022-01-07)
https://issues.apache.org/jira/browse/BEAM-13603: Shared object does not 
read from cache when using tag (created 2022-01-05)
https://issues.apache.org/jira/browse/BEAM-13598: Install Java 17 on 
Jenkins VM (created 2022-01-04)
https://issues.apache.org/jira/browse/BEAM-13582: Beam website precommit 
mentions broken links, but passes. (created 2021-12-30)
https://issues.apache.org/jira/browse/BEAM-13579: Cannot run 
python_xlang_kafka_taxi_dataflow validation script on 2.35.0 (created 
2021-12-29)
https://issues.apache.org/jira/browse/BEAM-13522: Spark tests failing 
PerKeyOrderingTest (created 2021-12-22)
https://issues.apache.org/jira/browse/BEAM-13504: Remove 
provided/compileOnly deps not intended for external use (created 2021-12-21)
https://issues.apache.org/jira/browse/BEAM-13503: BulkIO public 
constructor: Missing required property: throwWriteErrors (created 2021-12-21)
https://issues.apache.org/jira/browse/BEAM-13430: Upgrade Gradle version to 
7.3 (created 2021-12-09)
https://issues.apache.org/jira/browse/BEAM-13393: GroupIntoBatchesTest is 
failing (created 2021-12-07)
https://issues.apache.org/jira/browse/BEAM-13367: 
[beam_PostCommit_Python36] [ 
apache_beam.io.gcp.experimental.spannerio_read_it_test] Failure summary 
(created 2021-12-01)
https://issues.apache.org/jira/browse/BEAM-13314: Revise recommendations to 
manage Python pipeline dependencies.  (created 2021-11-24)
https://issues.apache.org/jira/browse/BEAM-13237: 
org.apache.beam.sdk.transforms.CombineTest$WindowingTests.testWindowedCombineGloballyAsSingletonView
 flaky on Dataflow Runner V2 (created 2021-11-12)
https://issues.apache.org/jira/browse/BEAM-13203: Potential data loss when 
using SnsIO.writeAsync (created 2021-11-08)
https://issues.apache.org/jira/browse/BEAM-13164: Race between member 
variable being accessed due to leaking uninitialized state via 
OutboundObserverFactory (created 2021-11-01)
https://issues.apache.org/jira/browse/BEAM-13087: 
apache_beam.runners.portability.fn_api_runner.translations_test.TranslationsTest.test_run_packable_combine_globally
 'apache_beam.coders.coder_impl._AbstractIterable' object is not reversible 
(created 2021-10-20)
https://issues.apache.org/jira/browse/BEAM-13078: Python DirectRunner does 
not emit data at GC time (created 2021-10-18)
https://issues.apache.org/jira/browse/BEAM-13076: Python AfterAny, AfterAll 
do not follow spec (created 2021-10-18)
https://issues.apache.org/jira/browse/BEAM-13059: Migrate GKE workloads to 
Containerd (created 2021-10-15)
https://issues.apache.org/jira/browse/BEAM-13058: Upgrade Kubernetes APIs 
(created 2021-10-15)
https://issues.apache.org/jira/browse/BEAM-13010: Delete orphaned files 
(created 2021-10-06)
https://issues.apache.org/jira/browse/BEAM-12995: Consumer group with 
random prefix (created 2021-10-04)
https://issues.apache.org/jira/browse/BEAM-12959: Dataflow error in 
CombinePerKey operation (created 2021-09-26)
https://issues.apache.org/jira/browse/BEAM-12867: Either Create or 
DirectRunner fails to produce all elements to the following transform (created 
2021-09-09)
https://issues.apache.org/jira/browse/BEAM-12843: (Broken Pipe induced) 
Bricked Dataflow Pipeline  (created 2021-09-06)
https://issues.apache.org/jira/browse/BEAM-12807: Java creates an incorrect 
pipeline proto when core-construction-java jar is not in the CLASSPATH (created 
2021-08-26)
https://issues.apache.org/jira/browse/BEAM-12792: Beam worker only installs 
--extra_package once (created 2021-08-24)
https://issues.apache.org/jira/browse/BEAM-12632: ElasticsearchIO: Enabling 
both User/Pass auth and SSL overwrites User/Pass (created 2021-07-16)
https://issues.apache.org/jira/browse/BEAM-12525: SDF BoundedSource seems 
to execute significantly slower than 'normal' BoundedSource (created 2021-06-22)
https://issues.apache.org/jira/browse/BEAM-12505: codecov/patch has poor 
behavior (created 2021-06-17)

P0 (outage) report

[Beam Jira Bot]

This is your daily summary of Beam's current outages. See 
https://beam.apache.org/contribute/jira-priorities/#p0-outage for the meaning 
and expectations around P0 issues.

BEAM-13213: OnWindowExpiration does not work without other state 
(https://issues.apache.org/jira/browse/BEAM-13213)

Re: Preparing for vendored dependency updates

[Kenneth Knowles]

Great. Thanks a ton for doing so much releasing!

On Tue, Jan 11, 2022 at 7:37 PM Luke Cwik  wrote:

> +1
>
> On Tue, Jan 11, 2022 at 5:32 PM Emily Ye  wrote:
>
>> Hi dev@,
>>
>> We will be releasing two updates to vendored packages, upgrading to gRPC
>> 1.43.2 and upgrading calcite 1.28.0 to use protobuf-java 3.19.2 (see
>> related issue [1]). We are aiming to release these as part of Beam 2.36.0.
>>
>> I realize I sent the gRPC vote release out already, but I've
>> self-nominated to do these vendored dependency releases and if there are no
>> objections, I'll send out the calcite vote once artifacts have been staged.
>>
>> Thank you!
>> Emily Ye
>>
>> [1] https://issues.apache.org/jira/browse/BEAM-13616
>>
>