BEAM-9330

[Cristian Constantinescu]

Hello,

I would like to contribute to
https://issues.apache.org/jira/browse/BEAM-9330, getting Beam to support
PROTOBUF and JSON schemas with Confluent Schema Registry.

I'm a bit of a Java newb (C# and Go are more up my alley) and this would be
my first contribution to any ASF projects. As such, if anyone a bit more
experienced is interested in a bit of pair programming with me or has time
for a bit of hand holding, I'd really appreciate it.

Thanks!

Re: Contributor permission for Beam Jira tickets

[Luke Cwik]

Welcome to the community.

It looks like someone has already added you.

Check out the contribution guide[1].

1: https://beam.apache.org/contribute/

On Fri, Aug 28, 2020 at 12:03 PM Omkar Deshpande 
wrote:

> Hi, my name is Omkar Deshpande. I am interested in contributing java kafka
> io module in the Apache Beam SDK. I'd like to be added as a Jira
> contributor so that I can assign issues to myself. My ASF Jira Username is
> omkardeshpande8.
>
> I don't see the email I sent earlier from work email in the archives.
> Sorry in advance, if this is a duplicate email.
>
> Omkar
>

Contributor permission for Beam Jira tickets

[Omkar Deshpande]

Hi, my name is Omkar Deshpande. I am interested in contributing java kafka
io module in the Apache Beam SDK. I'd like to be added as a Jira
contributor so that I can assign issues to myself. My ASF Jira Username is
omkardeshpande8.

I don't see the email I sent earlier from work email in the archives.
Sorry in advance, if this is a duplicate email.

Omkar

Re: Beam Dependency Check Report (2020-08-03)

[Tobiasz Kędzierski]

I created PR [1] with the fix. PTAL

[1] https://github.com/apache/beam/pull/12716

On Fri, Aug 28, 2020 at 10:20 AM Tobiasz Kędzierski <
tobiasz.kedzier...@polidea.com> wrote:

> HI,
>
> I created JIRA [1] for this issue.
> I will try to fix it.
>
> [1] https://issues.apache.org/jira/browse/BEAM-10831
>
> On Mon, Aug 3, 2020 at 7:18 PM Damian Gadomski <
> damian.gadom...@polidea.com> wrote:
>
>> That's probably caused by this [1] PR, workspace had been deleted before
>> the email was sent.
>>
>> +Udi Meiri  Moving the workspace clean up to the very
>> end of post-build actions should help.
>>
>> [1] https://github.com/apache/beam/pull/12326
>>
>> On Mon, Aug 3, 2020 at 5:42 PM Brian Hulette  wrote:
>>
>>> Does anyone know what went wrong here? It looks like the
>>> associated jenkins job [1] succeeded, and produced
>>> beam-dependency-check-report.html
>>>
>>> [1] https://ci-beam.apache.org/job/beam_Dependency_Check/279/
>>>
>>> On Mon, Aug 3, 2020 at 5:28 AM Apache Jenkins Server <
>>> jenk...@builds.apache.org> wrote:
>>>
 ERROR: File
 'src/build/dependencyUpdates/beam-dependency-check-report.html' does not
 exist
>>>
>>>

-- 

Tobiasz Kędzierski
Polidea  | Junior Software Engineer

E: tobiasz.kedzier...@polidea.com
[image: Polidea] 

Check out our projects! 
[image: Github]  [image: Facebook]
 [image: Twitter]
 [image: Linkedin]
 [image: Instagram]
 [image: Behance]
 [image: dribbble]

Beam Dependency Check Report (2020-08-28)

[Apache Jenkins Server]

High Priority Dependency Updates Of Beam Python SDK:


  Dependency Name
  Current Version
  Latest Version
  Release Date Of the Current Used Version
  Release Date Of The Latest Release
  JIRA Issue
  
cachetools
3.1.1
4.1.1
2019-12-23
2020-07-08BEAM-9017
chromedriver-binary
83.0.4103.39.0
85.0.4183.87.0
2020-07-08
2020-08-28BEAM-10426
fastavro
0.23.6
1.0.0.post1
2020-08-03
2020-08-28BEAM-10798
mock
2.0.0
3.0.5
2019-05-20
2019-05-20BEAM-7369
mypy-protobuf
1.18
1.23
2020-03-24
2020-06-29BEAM-10346
oauth2client
3.0.0
4.1.3
2018-12-10
2018-12-10BEAM-6089
pyarrow
0.17.1
1.0.1
2020-07-27
2020-08-24BEAM-10582
PyHamcrest
1.10.1
2.0.2
2020-01-20
2020-07-08BEAM-9155
pytest
4.6.11
6.0.1
2020-07-08
2020-08-03BEAM-8606
pytest-xdist
1.34.0
2.1.0
2020-08-17
2020-08-28BEAM-10713
tenacity
5.1.5
6.2.0
2019-11-11
2020-06-29BEAM-8607
High Priority Dependency Updates Of Beam Java SDK:


  Dependency Name
  Current Version
  Latest Version
  Release Date Of the Current Used Version
  Release Date Of The Latest Release
  JIRA Issue
  
com.amazonaws:amazon-kinesis-producer
0.13.1
0.14.1
2019-07-31
2020-07-31BEAM-10628
com.azure:azure-storage-blob
12.1.0
12.8.0
2019-12-05
2020-08-13BEAM-10800
com.datastax.cassandra:cassandra-driver-core
3.8.0
4.0.0
2019-10-29
2019-03-18BEAM-8674
com.esotericsoftware:kryo
4.0.2
5.0.0-RC9
2018-03-20
2020-08-14BEAM-5809
com.esotericsoftware.kryo:kryo
2.21
2.24.0
2013-02-27
2014-05-04BEAM-5574
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin
0.20.0
0.29.0
2019-02-11
2020-07-20BEAM-6645
com.google.api:gax
1.54.0
1.58.2
2020-02-27
2020-08-07BEAM-10348
com.google.api:gax-grpc
1.54.0
1.58.2
2020-02-27
2020-08-07BEAM-8676
com.google.api.grpc:grpc-google-cloud-pubsub-v1
1.85.1
1.90.1
2020-03-09
2020-08-04BEAM-8677
com.google.api.grpc:grpc-google-common-protos
1.12.0
1.18.1
2018-06-29
2020-08-11BEAM-8633
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1
0.85.1
0.105.0
2020-01-08
2020-08-21BEAM-8678
com.google.api.grpc:proto-google-cloud-bigtable-v2
1.9.1
1.14.0
2020-01-10
2020-07-21BEAM-8679
com.google.api.grpc:proto-google-cloud-pubsub-v1
1.85.1
1.90.1
2020-03-09
2020-08-04BEAM-8681
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1
1.49.1
1.60.0
2020-01-28
2020-08-26BEAM-8682
com.google.apis:google-api-services-bigquery
v2-rev20200719-1.30.10
v2-rev20200818-1.30.10
2020-07-26
2020-08-27BEAM-8684
com.google.apis:google-api-services-clouddebugger
v2-rev20200501-1.30.10
v2-rev20200807-1.30.10
2020-07-14
2020-08-17BEAM-8750
com.google.apis:google-api-services-cloudresourcemanager
v1-rev20200720-1.30.10
v2-rev20200810-1.30.10
2020-07-25
2020-08-14BEAM-8751
com.google.apis:google-api-services-dataflow
v1b3-rev20200713-1.30.10
v1beta3-rev12-1.20.0
2020-07-25
2015-04-29BEAM-8752
com.google.apis:google-api-services-healthcare
v1beta1-rev20200713-1.30.10
v1-rev20200819-1.30.10
2020-07-24
2020-08-26BEAM-10349
com.google.apis:google-api-services-pubsub
v1-rev20200713-1.30.10
v1-rev20200807-1.30.10
2020-07-25
2020-08-14BEAM-8753
com.google.apis:google-api-services-storage
v1-rev20200611-1.30.10
v1-rev20200727-1.30.10
2020-07-10
2020-08-06BEAM-8754
com.google.auto.service:auto-service
1.0-rc6
1.0-rc7
2019-07-16
2020-05-13BEAM-5541
com.google.auto.service:auto-service-annotations
1.0-rc6
1.0-rc7
2019-07-16
2020-05-13BEAM-10350
com.google.cloud:google-cloud-bigquery
1.108.0
1.117.0
2020-02-28
2020-08-25BEAM-8687
com.google.cloud:google-cloud-bigquerystorage
0.125.0-beta
1.5.0
2020-02-20

Beam Dependency Check Report (2020-08-28)

[Apache Jenkins Server]

ERROR: File 'src/build/dependencyUpdates/beam-dependency-check-report.html' does not exist

Re: Adding new Jira component for Twister2

[Pulasthi Supun Wickramasinghe]

Hi Ismaël

Yes, just checked and used that for the Jira's. Thanks

Best Regards,
Pulasthi

On Fri, Aug 28, 2020 at 10:59 AM Ismaël Mejía  wrote:

> Component created I forgot to do this when we merged, please confirm
> if it works.
>
> On Fri, Aug 28, 2020 at 7:47 AM Pulasthi Supun Wickramasinghe
>  wrote:
> >
> > Hi All,
> >
> > While creating a issue for Twister2 i noticed currently there is no
> component tag for the Twister2 runner. Should we add a new component
> "runner-twister2", if so what are the steps to creating a component?
> >
> > Best Regards,
> > Pulasthi
> >
> > --
> > Pulasthi S. Wickramasinghe
> > PhD Candidate  | Research Assistant
> > School of Informatics and Computing | Digital Science Center
> > Indiana University, Bloomington
> > cell: 224-386-9035
>


-- 
Pulasthi S. Wickramasinghe
PhD Candidate  | Research Assistant
School of Informatics and Computing | Digital Science Center
Indiana University, Bloomington
cell: 224-386-9035

Re: Contributor permission for Beam Jira tickets

[Ismaël Mejía]

Hello Omkar I added you as a contributor you can now self assign
tickets. I also assigned BEAM-10829 to you.
Welcome to Beam, Enjoy!

On Fri, Aug 28, 2020 at 5:49 AM Deshpande, Omkar
 wrote:
>
> Hello,
>
> My name is Omkar Deshpande. I would like to update java kafkaio module in 
> beam to support writing headers.
>
> Can someone add me as a contributor for Beam's Jira issue tracker?
>
> I would like to contribute to this Jira : 
> https://issues.apache.org/jira/browse/BEAM-10829
>
>
> Jira username:omkardeshpande8
>
> Omkar
>

Re: Adding new Jira component for Twister2

[Ismaël Mejía]

Component created I forgot to do this when we merged, please confirm
if it works.

On Fri, Aug 28, 2020 at 7:47 AM Pulasthi Supun Wickramasinghe
 wrote:
>
> Hi All,
>
> While creating a issue for Twister2 i noticed currently there is no component 
> tag for the Twister2 runner. Should we add a new component "runner-twister2", 
> if so what are the steps to creating a component?
>
> Best Regards,
> Pulasthi
>
> --
> Pulasthi S. Wickramasinghe
> PhD Candidate  | Research Assistant
> School of Informatics and Computing | Digital Science Center
> Indiana University, Bloomington
> cell: 224-386-9035

Beam Dependency Check Report (2020-08-28)

[Apache Jenkins Server]

ERROR: File 'src/build/dependencyUpdates/beam-dependency-check-report.html' does not exist

Re: [DISCUSS][BEAM-10670] Migrating BoundedSource/UnboundedSource to execute as a Splittable DoFn for non-portable Java runners

[Maximilian Michels]

Thanks Luke! I've had a pass.

-Max

On 28.08.20 01:22, Luke Cwik wrote:

As an update.

Direct and Twister2 are done.
Samza: is ready for review[1].
Flink: is almost ready for review. [2] lays all the groundwork for the 
migration and [3] finishes the migration (there is a timeout happening 
in FlinkSubmissionTest that I'm trying to figure out).

No further updates on Spark[4] or Jet[5].

@Maximilian Michels  or @t...@apache.org 
, can either of you take a look at the 
Flink PRs?
@ke.wu...@icloud.com , Since Xinyu delegated 
to you, can you take another look at the Samza PR?


1: https://github.com/apache/beam/pull/12617
2: https://github.com/apache/beam/pull/12706
3: https://github.com/apache/beam/pull/12708
4: https://github.com/apache/beam/pull/12603
5: https://github.com/apache/beam/pull/12616

On Tue, Aug 18, 2020 at 11:42 AM Pulasthi Supun Wickramasinghe 
mailto:pulasthi...@gmail.com>> wrote:


Hi Luke

Will take a look at this as soon as possible and get back to you.

Best Regards,
Pulasthi

On Tue, Aug 18, 2020 at 2:30 PM Luke Cwik mailto:lc...@google.com>> wrote:

I have made some good progress here and have gotten to the
following state for non-portable runners:

DirectRunner[1]: Merged. Supports Read.Bounded and Read.Unbounded.
Twister2[2]: Ready for review. Supports Read.Bounded, the
current runner doesn't support unbounded pipelines.
Spark[3]: WIP. Supports Read.Bounded, Nexmark suite passes. Not
certain about level of unbounded pipeline support coverage since
Spark uses its own tiny suite of tests to get unbounded pipeline
coverage instead of the validates runner set.
Jet[4]: WIP. Supports Read.Bounded. Read.Unbounded definitely
needs additional work.
Sazma[5]: WIP. Supports Read.Bounded. Not certain about level of
unbounded pipeline support coverage since Spark uses its own
tiny suite of tests to get unbounded pipeline coverage instead
of the validates runner set.
Flink: Unstarted.

@Pulasthi Supun Wickramasinghe  ,
can you help me with the Twister2 PR[2]?
@Ismaël Mejía , is PR[3] the expected
level of support for unbounded pipelines and hence ready for review?
@Jozsef Bartok , can you help me out
to get support for unbounded splittable DoFn's into Jet[4]?
@Xinyu Liu , is PR[5] the expected
level of support for unbounded pipelines and hence ready for review?

1: https://github.com/apache/beam/pull/12519
2: https://github.com/apache/beam/pull/12594
3: https://github.com/apache/beam/pull/12603
4: https://github.com/apache/beam/pull/12616
5: https://github.com/apache/beam/pull/12617

On Tue, Aug 11, 2020 at 10:55 AM Luke Cwik mailto:lc...@google.com>> wrote:

There shouldn't be any changes required since the wrapper
will smoothly transition the execution to be run as an SDF.
New IOs should strongly prefer to use SDF since it should be
simpler to write and will be more flexible but they can use
the "*Source"-based APIs. Eventually we'll deprecate the
APIs but we will never stop supporting them. Eventually they
should all be migrated to use SDF and if there is another
major Beam version, we'll finally be able to remove them.

On Tue, Aug 11, 2020 at 8:40 AM Alexey Romanenko
mailto:aromanenko@gmail.com>>
wrote:

Hi Luke,

Great to hear about such progress on this!

Talking about opt-out for all runners in the future,
will it require any code changes for current
“*Source”-based IOs or the wrappers should completely
smooth this transition?
Do we need to require to create new IOs only based on
SDF or again, the wrappers should help to avoid this?


On 10 Aug 2020, at 22:59, Luke Cwik mailto:lc...@google.com>> wrote:

In the past couple of months wrappers[1, 2] have been
added to the Beam Java SDK which can execute
BoundedSource and UnboundedSource as Splittable DoFns.
These have been opt-out for portable pipelines (e.g.
Dataflow runner v2, XLang pipelines on Flink/Spark)
and opt-in using an experiment for all other pipelines.

I would like to start making the non-portable
pipelines starting with the DirectRunner[3] to be
opt-out with the plan that eventually all runners will
only execute splittable DoFns and the
  

Re: Beam Dependency Check Report (2020-08-03)

[Tobiasz Kędzierski]

HI,

I created JIRA [1] for this issue.
I will try to fix it.

[1] https://issues.apache.org/jira/browse/BEAM-10831

On Mon, Aug 3, 2020 at 7:18 PM Damian Gadomski 
wrote:

> That's probably caused by this [1] PR, workspace had been deleted before
> the email was sent.
>
> +Udi Meiri  Moving the workspace clean up to the very
> end of post-build actions should help.
>
> [1] https://github.com/apache/beam/pull/12326
>
> On Mon, Aug 3, 2020 at 5:42 PM Brian Hulette  wrote:
>
>> Does anyone know what went wrong here? It looks like the
>> associated jenkins job [1] succeeded, and produced
>> beam-dependency-check-report.html
>>
>> [1] https://ci-beam.apache.org/job/beam_Dependency_Check/279/
>>
>> On Mon, Aug 3, 2020 at 5:28 AM Apache Jenkins Server <
>> jenk...@builds.apache.org> wrote:
>>
>>> ERROR: File
>>> 'src/build/dependencyUpdates/beam-dependency-check-report.html' does not
>>> exist
>>
>>

Contributor permission for Beam Jira tickets

[Deshpande, Omkar]

Hello,


My name is Omkar Deshpande. I would like to update java kafkaio module in beam 
to support writing headers.

Can someone add me as a contributor for Beam's Jira issue tracker?

I would like to contribute to this Jira : 
https://issues.apache.org/jira/browse/BEAM-10829

Jira username:omkardeshpande8

Omkar