I believe KinesisIO only works with Kinesis Data Streams.
> On Jun 8, 2020, at 9:42 AM, Brahmachaitanya Wajapey
> wrote:
>
> Hi,
>
> Apache beam supports KinesisIO connector. Is this only for Kinesis streams or
> it can be used to put records in Kinesis firehose as well?
>
> I have a
DirectRunner is not well-tested for xlang transforms and you need to
specify jar_packages experimental flag for Java dependencies from Python
SDK. I'd recommend using 2.22 + FlinkRunner for xlang pipelines.
On Mon, Jun 8, 2020 at 3:27 PM Chamikara Jayalath
wrote:
> To clarify, Kafka dependency
Thank you Brian!
Something a little for all of us to celebrate. Beam 2.5.0 was released on
2018/06/21. This is the 17th release after that in less than 24 months. We
were really great at releasing with an average velocity of 8 releases/year.
On Mon, Jun 8, 2020 at 11:30 AM Kyle Weaver wrote:
>
We are using the Beam 2.20 Python SDK on a Flink 1.9 runner. Our
messages originate from a custom source that consumes messages from a
Kafka topic and emits them in the order of their Kafka offsets to a
DoFn. After this DoFn processes the messages, they are emitted to a
custom sink that sends
To clarify, Kafka dependency was already available as an embedded
dependency in Java SDK Harness but not sure if this worked for
DirectRunner. starting 2.22 we'll be staging dependencies from the
environment during pipeline submission.
On Mon, Jun 8, 2020 at 3:23 PM Chamikara Jayalath
wrote:
>
Seems like Java dependency is not being properly set up when running the
cross-language Kafka step. I don't think this was available for Beam 2.21.
Can you try with the latest Beam HEAD or Beam 2.22 when it's released ?
+Heejong Lee
On Mon, Jun 8, 2020 at 12:39 PM Piotr Filipiuk
wrote:
>
On Mon, Jun 8, 2020 at 2:06 PM Chad Dombrova wrote:
> Even when running portably, Dataflow still has its own implementation of
>> PubSubIO that is switched out for Python's "implementation." (It's actually
>> built into the same layer that provides the shuffle/group-by-key
>> implementation.)
> Even when running portably, Dataflow still has its own implementation of
> PubSubIO that is switched out for Python's "implementation." (It's actually
> built into the same layer that provides the shuffle/group-by-key
> implementation.) However, if you used the external Java PubSubIO it may not
On Mon, Jun 8, 2020 at 12:57 PM Chad Dombrova wrote:
> Hi all,
> quick followup question:
>
>
>> small correction. While the new runner will be available with Beam 2.21,
>>> the Cross-Language support will be available in 2.22.
>>> There will be limitations in the initial set of connectors you
Hi all,
quick followup question:
> small correction. While the new runner will be available with Beam 2.21,
>> the Cross-Language support will be available in 2.22.
>> There will be limitations in the initial set of connectors you can use
>> with Cross-Lang. But at least you will have something
Pasting the error inline:
ERROR:root:severity: ERROR
timestamp {
seconds: 1591405163
nanos: 81500
}
message: "Client failed to dequeue and process the value"
trace: "org.apache.beam.sdk.util.UserCodeException:
java.lang.NoClassDefFoundError:
Hooray! Thanks for shepherding the release Brian :)
On Mon, Jun 8, 2020 at 11:24 AM Brian Hulette wrote:
> I'm happy to announce that we have approved the 2.22.0 release.
>
> There are 5 approving votes, 3 of which are binding:
> * Robert Bradshaw
> * Ahmet Altay
> * Pablo Estrada
>
> Thanks
I'm happy to announce that we have approved the 2.22.0 release.
There are 5 approving votes, 3 of which are binding:
* Robert Bradshaw
* Ahmet Altay
* Pablo Estrada
Thanks for everyone's help! I'm going to finalize the release and send out
the official release announcement tomorrow.
Brian
Tyson, Brian - thanks for details!
> On 8 Jun 2020, at 18:55, Brian Hulette wrote:
>
> Sometimes triggering with a key sentence starts a job based on an earlier
> commit in the PR branch, and repeatedly trying to trigger can start several
> jobs validating that same incorrect commit. You can
Hello Aman,
Thank you for the introduction and your interest to work on Apache Beam
documentation with Season of Docs. To participate in the program you need
to follow the guides here [1] [2]. If you are new to the program, we
suggest:
Start by studying our proposed project ideas and expected
Sometimes triggering with a key sentence starts a job based on an earlier
commit in the PR branch, and repeatedly trying to trigger can start several
jobs validating that same incorrect commit. You can see a yellow circle
next to the old commit ID when this happens. On the day of 2.22.0 cut Pablo
Respected Sir,
I am Aman Singal, I am a 2nd Year undergraduate student. I really would
like to contribute in your esteemed organization in writing documentation.
I have read and understood the requirement of the project "Update of the
runner comparison page / capability matrix". I request you to
Hi,
Apache beam supports KinesisIO connector. Is this only for Kinesis streams or
it can be used to put records in Kinesis firehose as well?
I have a requirement where i read events from kafka and I have to put the
records in kinesis firehose.
So, can we use KinesisIO connector for this?
There is an active effort in coordination with the Apache Infrastructure
team to move Beam to a new dedicated Jenkins cluster. This will resolve the
issues described and add some new functionality as well within the next
month.
On Mon, Jun 8, 2020 at 6:29 AM Alexey Romanenko
wrote:
> Yes, the
Welcome Zijie!
I've added you as contributor.
Best
-P.
On Sat, Jun 6, 2020 at 2:03 PM Zijie Song wrote:
> Hi,
>
> My name is Zijie Song and I am currently an SWE Intern at Google. I will
> work on the Apache Beam Project for this summer. So, I would like to be
> added as a contributor in the
Yes, the key sentences still work. The problem that it triggers jobs in non
deterministic way. Sometimes, to run a job, I need to type the same command
twice or more times.
> On 8 Jun 2020, at 14:25, Jean-Baptiste Onofre wrote:
>
> Hi,
>
>
> Actually, it’s because of the PR plugin we are
Hi,
Actually, it’s because of the PR plugin we are using on Jenkins.
We have the choice between two plugins:
- the "old/deprecated" one that allow key sentence like "retest this please".
I’m still using it on projects like Karaf just for that ;)
- the "new" plugin just trigger build when the
Hello,
I think we have an issue, that seems to happen from time to time, to trigger
the Jenkins jobs from PRs.
For me, usually, I need to type and enter 2 or even more times the same command
as a PR comment (like ‘retest this please’ or at least 'Run Java PreCommit’) to
see that it finally
ERROR: File 'src/build/dependencyUpdates/beam-dependency-check-report.html' does not exist
24 matches
Mail list logo
