Re: [DISCUSS] Apache Beam 2.1.0 release next week ?

[Jean-Baptiste Onofré]

Hi Stephen,

Thanks for the update.

I have an issue on my machine with SpannerIOTest. I will create the release 
branch as soon as this is fix. Then, we will be able to cherry-pick the fix we want.


I keep you posted.

Regards
JB

On 06/28/2017 09:37 PM, Stephen Sisk wrote:

hi!

I'm hopeful we can get the fix for BEAM-2533 into this release as well,
there's a bigtable fix in the next version that'd be good to have. The
bigtable client release should be in the next day or two.

S

On Mon, Jun 26, 2017 at 12:03 PM Jean-Baptiste Onofré 
wrote:


Hi guys,

just a quick update about the 2.1.0 release.

I will complete the Jira triage tomorrow.

I plan to create the release branch Wednesday.

Thanks !
Regards
JB

On 06/22/2017 04:23 AM, Jean-Baptiste Onofré wrote:

Hi guys,

As we released 2.0.0 (first stable release) last month during ApacheCon,

and to

maintain our release pace, I would like to release 2.1.0 next week.

This release would include lot of bug fixes and some new features:

https://issues.apache.org/jira/projects/BEAM/versions/12340528

I'm volunteer to be release manager for this one.

Thoughts ?

Thanks,
Regards
JB


--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com





--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

[Proposal] Fine-grained Resource Configuration in Beam

[Pei HE]

Hi guys,
We are using Blink runner (fork of Flink) and an internal MapReduce-ish
runner.

We want to configure resources (parallelism, CPU, Memory, e.t.c) in an
unified way cross runners.

Haozhi and I have drafted a proposal, and would like to have feedback from
the community.

https://docs.google.com/document/d/1N0y64dbzmukLLEy6M9CygdI_H88pIS3NtcOAkL5-oVw/edit#

Thanks
--
Pei

Re: KafkaIO, Warning on offset gap

[Raghu Angadi]

Fixing it in https://github.com/apache/beam/pull/3461.

Thanks for reporting the issue.

On Wed, Jun 28, 2017 at 8:37 AM, Raghu Angadi  wrote:

> Hi Elmar,
>
> You are right. We should not log this at all when the gaps are expected as
> you pointed out. I don't think client can check if compaction is enabled
> for a topic through Consumer api.
>
> I think we should remove the log. The user can't really act on it other
> than reporting it. I will send a PR.
>
> As a temporary work around you can disable logging for a particular class
> on the worker with --workerLogLevelOverrides
>  option. But this
> this would suppress rest of the logging the reader.
>
> Raghu
>
>
> On Wed, Jun 28, 2017 at 4:12 AM, Elmar Weber  wrote:
>
>> Hello,
>>
>> I'm testing the KafkaIO with Google Cloud dataflow and getting warnings
>> when working with compacted logs. In the code there is a relevant check:
>>
>>
>> https://github.com/apache/beam/blob/master/sdks/java/io/kafk
>> a/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L1158
>>
>> // sanity check
>> if (offset != expected) {
>>   LOG.warn("{}: gap in offsets for {} at {}. {} records missing.",
>>   this, pState.topicPartition, expected, offset - expected);
>> }
>>
>> From what I understand, this can happen when log compaction is enabled
>> because the relevant entry can get cleaned up by Kafka with a newer one.
>> In this case, shouldn't this be a info log and / or warn only when log
>> compaction is disabled for the topic?
>>
>>
>> I'm still debugging some stuff because the pipeline also stops reading on
>> compacted logs, I'm not sure if this related / could also be an issue with
>> my Kafka test installation, but as far as I understand the gaps are
>> expected behaviour with log compaction enabled.
>>
>> Thanks,
>> Elmar
>>
>>
>

Re: [DISCUSS] Apache Beam 2.1.0 release next week ?

[Stephen Sisk]

hi!

I'm hopeful we can get the fix for BEAM-2533 into this release as well,
there's a bigtable fix in the next version that'd be good to have. The
bigtable client release should be in the next day or two.

S

On Mon, Jun 26, 2017 at 12:03 PM Jean-Baptiste Onofré 
wrote:

> Hi guys,
>
> just a quick update about the 2.1.0 release.
>
> I will complete the Jira triage tomorrow.
>
> I plan to create the release branch Wednesday.
>
> Thanks !
> Regards
> JB
>
> On 06/22/2017 04:23 AM, Jean-Baptiste Onofré wrote:
> > Hi guys,
> >
> > As we released 2.0.0 (first stable release) last month during ApacheCon,
> and to
> > maintain our release pace, I would like to release 2.1.0 next week.
> >
> > This release would include lot of bug fixes and some new features:
> >
> > https://issues.apache.org/jira/projects/BEAM/versions/12340528
> >
> > I'm volunteer to be release manager for this one.
> >
> > Thoughts ?
> >
> > Thanks,
> > Regards
> > JB
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: KafkaIO, Warning on offset gap

[Raghu Angadi]

Hi Elmar,

You are right. We should not log this at all when the gaps are expected as
you pointed out. I don't think client can check if compaction is enabled
for a topic through Consumer api.

I think we should remove the log. The user can't really act on it other
than reporting it. I will send a PR.

As a temporary work around you can disable logging for a particular class
on the worker with --workerLogLevelOverrides
 option. But this this
would suppress rest of the logging the reader.

Raghu


On Wed, Jun 28, 2017 at 4:12 AM, Elmar Weber  wrote:

> Hello,
>
> I'm testing the KafkaIO with Google Cloud dataflow and getting warnings
> when working with compacted logs. In the code there is a relevant check:
>
>
> https://github.com/apache/beam/blob/master/sdks/java/io/kafk
> a/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L1158
>
> // sanity check
> if (offset != expected) {
>   LOG.warn("{}: gap in offsets for {} at {}. {} records missing.",
>   this, pState.topicPartition, expected, offset - expected);
> }
>
> From what I understand, this can happen when log compaction is enabled
> because the relevant entry can get cleaned up by Kafka with a newer one.
> In this case, shouldn't this be a info log and / or warn only when log
> compaction is disabled for the topic?
>
>
> I'm still debugging some stuff because the pipeline also stops reading on
> compacted logs, I'm not sure if this related / could also be an issue with
> my Kafka test installation, but as far as I understand the gaps are
> expected behaviour with log compaction enabled.
>
> Thanks,
> Elmar
>
>

KafkaIO, Warning on offset gap

[Elmar Weber]

Hello,

I'm testing the KafkaIO with Google Cloud dataflow and getting warnings 
when working with compacted logs. In the code there is a relevant check:



https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L1158

// sanity check
if (offset != expected) {
  LOG.warn("{}: gap in offsets for {} at {}. {} records missing.",
  this, pState.topicPartition, expected, offset - expected);
}

From what I understand, this can happen when log compaction is enabled 
because the relevant entry can get cleaned up by Kafka with a newer one.
In this case, shouldn't this be a info log and / or warn only when log 
compaction is disabled for the topic?



I'm still debugging some stuff because the pipeline also stops reading 
on compacted logs, I'm not sure if this related / could also be an issue 
with my Kafka test installation, but as far as I understand the gaps are 
expected behaviour with log compaction enabled.


Thanks,
Elmar

Jenkins build is still unstable: beam_Release_NightlySnapshot #461

[Apache Jenkins Server]

See