Re: Specifying environment for cross-language transform expansion

I believe we could change that more or less the same as we can deprecate / stop supporting any other parameter of any method. If python starts to support natively Kafka IO, then we can simply log warning / raise exception (one after the other). That seems like natural development. Maybe I

Re: Specifying environment for cross-language transform expansion

I would absolutely understand this, if it would be mostly impossible or at least really hard to get the user friendly behavior. But we are mostly there in this case. When we can actually quite simply pass the supported environment via parameter, I think we should go for it. I have created a

Re: Specifying environment for cross-language transform expansion

On 6/29/21 11:04 PM, Robert Bradshaw wrote: You can configure the environment in the current state, you just have to run your own expansion service that has a different environment backed into it (or, makes this configurable). Yes, that is true. On the other hand that lacks some

Re: Specifying environment for cross-language transform expansion

Thanks for pointing to that thread. 1) I'm - as well as Kyle - fine with the approach that we use a "preferred environment" for the expansion service. We only need to pass it via command line. Yes, the command line might be generally SDK-dependent, and that makes it expansion dependent,

Re: [PROPOSAL] Vendored gRPC 1.36.0 0.2 Release

Sounds good. I imagine the tests are set up that way so that users can run tests locally on platforms that do not support epoll. Will the change in shaded deps preserve this? Kenn On Tue, Jun 29, 2021 at 12:10 PM Luke Cwik wrote: > I noticed that at some point in time we broke support for

Flaky test issue report (29)

This is your daily summary of Beam's current flaky tests (https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20labels%20%3D%20flake) These are P1 issues because they have a major negative impact on the community and make it hard to

Re: Specifying environment for cross-language transform expansion

I would slightly disagree that this breaks the black box nature of the expansion, the "how the transform expands" remains unknown to the SDK requesting the expansion, the "how the transform executes" - on the other hand - is something that the SDK must cooperate on - it knows (or could or

P1 issues report (42)

This is your daily summary of Beam's current P1 issues, not including flaky tests (https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20priority%20%3D%20P1%20AND%20(labels%20is%20EMPTY%20OR%20labels%20!%3D%20flake). See

Re: Specifying environment for cross-language transform expansion

The argument for being able to accept (possibly ordered list of) execution environments is in that this could make a single instance of execution service reusable by various clients with different requirements. Moreover, the two approaches are probably orthogonal - users could specify

Re: BEAM-8218_Implement Apache PulsarIO status?

Hi Marco, The Java SDK connector for Apache PulsarIO is very demanded but, unfortunately, I’m not aware if anyone works on this (based on Jira discussion). So, feel free to take it! One of the example, you may take a look on RabbitMqIO [1] that is quite straitforward in terms of

Re: Specifying environment for cross-language transform expansion

If I understand it correctly, there is currently no place to set the defaultEnvironmentType - python's KafkaIO uses either 'expansion_service' given by the user (which might be a host:port, or an object that has appropriate method), or calls 'default_io_expansion_service' - which in turn runs

Re: Specifying environment for cross-language transform expansion

I’m sorry if I missed something but do you mean that PortablePipelineOptions.setDefaultEnvironmentType(String) doesn’t work for you? Or it’s only a specific case while using portable KafkaIO? > On 29 Jun 2021, at 09:51, Jan Lukavský wrote: > > Hi, > > I have come across an issue with

Re: [VOTE] Release 2.31.0, release candidate #1

(non binding) +1, Approve the release >From [TFX](https://github.com/tensorflow/tfx): our automated integration tests >on Python 3.6 and Linux pass without any problem. On 2021/06/29 00:09:41, Andrew Pilloud wrote: > Hi everyone, > Please review and vote on the release candidate #1 for the

BEAM-8218_Implement Apache PulsarIO status?

Hi everyone! I am planning to start working on this ticket BEAM-8218 - Implement Apache PulsarIO Is there any example about what is about? or any progress on this topic? Thanks in advance, -- *Marco Robles* *|* WIZELINE Software Engineer

Specifying environment for cross-language transform expansion

Hi, I have come across an issue with cross-language transforms. My setup is I have working environment type PROCESS and I cannot use DOCKER. When I use Python's KafkaIO, it unfortunately - by default - expands to docker environment, which then fails due to missing 'docker' command. I didn't

Re: Beam Contributor Request

Hello Jack, You were added to the contributors group so you can self assign tickets. Welcome to Beam! Ismaël On Mon, Jun 28, 2021 at 9:35 PM Jack McCluskey wrote: > Hey everyone, > > My name is Jack McCluskey and I'd like to be added to the contributor > list. My Jira username is

Re: Apache Beam Contributor List Request

Hello Marco, You were added to the contributor group so you can now self-assign JIRA tickets. Welcome to Beam! Ismaël On Mon, Jun 28, 2021 at 6:58 PM Marco Robles Pulido wrote: > > Hi team, > > This is Marco and I would like to be added to the Apache Beam contributor > list, my username is

Specifying environment for cross-language transform expansion

Hi, I have come across an issue with cross-language transforms. My setup is I have working environment type PROCESS and I cannot use DOCKER. When I use Python's KafkaIO, it unfortunately - by default - expands to docker environment, which then fails due to missing 'docker' command. I didn't

Re: FileIO with custom sharding function

> How will @RequiresStableInput prevent this situation when running batch > use case? > So this is handled in combination of @RequiresStableInput and output file finalization. @RequiresStableInput (or Reshuffle for most runners) makes sure that the input provided for the write stage does not get