[GitHub] samza pull request #884: SAMZA-2073: Do not commit the task offsets when shu...

[shanthoosh]

GitHub user shanthoosh opened a pull request:

https://github.com/apache/samza/pull/884

SAMZA-2073: Do not commit the task offsets when shutting down the 
SamzaContainer.

SAMZA-1489 added support for committing the offsets of all the running 
tasks in a SamzaContainer. Other components in samza such as a 
CheckpointListener's  aren't developed to account for possibility of commit 
after the consumers are stopped.

This  unnecessarily results in a unclean shutdown of a samza standalone 
processor during the rebalancing phase. Here're the sample logs:
```
apache.samza.container.SamzaContainer@129d533c to shutdown.
2019/01/15 02:38:09.738 INFO [SamzaContainer] [Samza StreamProcessor 
Container Thread-0] [hello-brooklin-task] [] Shutting down SamzaContainer.
2019/01/15 02:38:09.738 INFO [SamzaContainer] [Samza StreamProcessor 
Container Thread-0] [hello-brooklin-task] [] Shutting down consumer multiplexer.
2019/01/15 02:38:09.741 INFO [KafkaProducer] 
[hello-brooklin-task-i001-auditor] [hello-brooklin-task] [] [Producer 
clientId=hello-brooklin-task-i001-auditor, transactionalId=nullClosing the 
Kafka producer with timeoutMillis = 9223370489334886066 ms.
2019/01/15 02:38:09.748 INFO [SamzaContainer] [Samza StreamProcessor 
Container Thread-0] [hello-brooklin-task] [] Shutting down task instance stream 
tasks.
2019/01/15 02:38:09.748 INFO [SamzaContainer] [Samza StreamProcessor 
Container Thread-0] [hello-brooklin-task] [] Shutting down timer executor
2019/01/15 02:38:09.749 INFO [SamzaContainer] [Samza StreamProcessor 
Container Thread-0] [hello-brooklin-task] [] Committing offsets for all task 
instances
2019/01/15 02:38:09.753 ERROR [SamzaContainer] [Samza StreamProcessor 
Container Thread-0] [hello-brooklin-task] [] Caught exception/error while 
shutting down container.
java.lang.IllegalStateException: This consumer has already been closed.
 at 
org.apache.kafka.clients.consumer.KafkaConsumer.acquireAndEnsureOpen(KafkaConsumer.java:1735)
 at 
org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1214)
 at 
com.linkedin.kafka.liclients.consumer.LiKafkaConsumerImpl.commitOffsets(LiKafkaConsumerImpl.java:311)
 at 
com.linkedin.kafka.liclients.consumer.LiKafkaConsumerImpl.commitSync(LiKafkaConsumerImpl.java:282)
 at 
com.linkedin.brooklin.client.BaseConsumerImpl.commit(BaseConsumerImpl.java:136)
```
 

 

Since the final commit is not critical, it will be better to not do it as a 
part of the SamzaContainer shutdown sequence. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shanthoosh/samza 
remove_task_commit_during_shutdown

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/884.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #884


commit 5cb3faf01e9a1bfd4f81fb0d927dc774c8d16dd4
Author: Shanthoosh Venkataraman 
Date:   2019-01-16T02:26:05Z

Do not commit the task offsets when the task instances are shutting down.




---

[GitHub] samza pull request #883: SAMZA-2072: Update guava to 23.0

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/883


---

[GitHub] samza pull request #883: SAMZA-2072: Update guava to 23.0

[xinyuiscool]

GitHub user xinyuiscool opened a pull request:

https://github.com/apache/samza/pull/883

SAMZA-2072: Update guava to 23.0

Startpoint is relying on an old version of guava, which should be updated 
to 23.0 for the newer api.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xinyuiscool/samza SAMZA-2072

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/883.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #883


commit 040fe613ccfe053f4ce41b54862a97d717c7031d
Author: xiliu 
Date:   2019-01-16T01:26:05Z

SAMZA-2072: Update guava to 23.0




---

[GitHub] samza pull request #882: DiagnosticsAppender for log4j2

[rmatharu]

GitHub user rmatharu opened a pull request:

https://github.com/apache/samza/pull/882

DiagnosticsAppender for log4j2

This PR adds a DiagnosticsAppender for log4j2. 
There exists one already for log4j in samza-log4j. 

However the Appender, LogEvent, and Configuration APIs between log4j2 and 
log4j are completely different. 
So there was very little overlap (and very little value) in creating a 
base-class for the two appenders. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rmatharu/samza log4j2appender

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/882.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #882


commit 0212d4dc40e80f07c05b0d7b0fdc2d0491f941d3
Author: Ray Matharu 
Date:   2019-01-16T01:14:29Z

Adding DiagnosticsAppender for log4j2




---

[GitHub] samza pull request #881: SAMZA-2068: Separating container launch logic into ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/881


---

[GitHub] samza pull request #873: Samza 2070 : Upgrade calcite and add tests for arra...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/873


---

[GitHub] samza pull request #881: SAMZA-2068: Separating container launch logic into ...

[xinyuiscool]

GitHub user xinyuiscool opened a pull request:

https://github.com/apache/samza/pull/881

SAMZA-2068: Separating container launch logic into util class

The container launch logic needs to be invoked for beam-runner to run beam 
containers. This is a small refactoring of LocalContainerRunner.java.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xinyuiscool/samza SAMZA-2068

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/881.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #881


commit 6c4f2b52b17836e89f10139dfa27ff5855d7fad9
Author: xiliu 
Date:   2019-01-15T20:54:36Z

SAMZA-2068: Separating container launch logic into util class




---

Re: [DISCUSS] Mandatory migration of Samza git repo to gitbox.apache.org

[Yi Pan]

+1 (binding) for the move as well.

@Jake Maes  I think our merge script requires setting
up a local alias 'apache-samza' for the remote repo. Hence, it should be a
simple step to point the 'apache-samza' repo to the new repo on gitbox,
hopefully.

-Yi

On Tue, Jan 15, 2019 at 12:12 PM Jake Maes  wrote:

> Yeah, sounds pretty light weight.
>
> Quick question for committers: Do we anticipate this affecting our merge
> script? Perhaps we won't need the merge script, since we'll be able to use
> github directly. Thoughts?
>
> -Jake
>
> On Tue, Jan 15, 2019 at 11:49 AM Prateek Maheshwari 
> wrote:
>
> > Thanks for starting the discussion Pawas. I'm +1 (binding) for the
> > migration.
> >
> > - Prateek
> >
> > On Tue, Jan 15, 2019 at 11:44 AM Pawas Chhokra 
> > wrote:
> > >
> > > Hi all,
> > >
> > > As mandated by the Apache Infrastructure Team, all git repositories
> must
> > be
> > > migrated from git-wip-us.apache.org URL to gitbox.apache.org, as the
> old
> > > service is being decommissioned. This needs to happen before February
> > 7th,
> > > and this ticket  is
> to
> > > check if migrating Samza on 11 AM, Jan 25, 2019 is acceptable to
> > everyone.
> > >
> > > Thanks & Regards,
> > > Pawas Chhokra
> >
>

Re: [ANNOUNCE] New committer announcement

[Daniel Nishimura]

Congrats Shanthoosh! Well deserved. 

Sent from my iPhone

> On Jan 15, 2019, at 12:13 PM, Jake Maes  wrote:
> 
> Congrats and welcome, Shanthoosh!
> 
>> On Tue, Jan 15, 2019 at 11:50 AM Xinyu Liu  wrote:
>> 
>> Hi, all,
>> 
>> Please join me and the rest of the Samza PMC in welcoming a new committer:
>> 
>> - Shanthoosh Venkataraman: of, but not limited to, his work on Samza
>> standalone.
>> 
>> Through his work on developing new features such as host affinity on
>> standalone and refining existing Samza code base, the PMC trusts Shanthoosh
>> with the responsibilities of a Samza committer.
>> 
>> Xinyu
>> 

Re: [ANNOUNCE] New committer announcement

[Jake Maes]

Congrats and welcome, Shanthoosh!

On Tue, Jan 15, 2019 at 11:50 AM Xinyu Liu  wrote:

> Hi, all,
>
> Please join me and the rest of the Samza PMC in welcoming a new committer:
>
>  - Shanthoosh Venkataraman: of, but not limited to, his work on Samza
> standalone.
>
> Through his work on developing new features such as host affinity on
> standalone and refining existing Samza code base, the PMC trusts Shanthoosh
> with the responsibilities of a Samza committer.
>
> Xinyu
>

Re: [DISCUSS] Mandatory migration of Samza git repo to gitbox.apache.org

[Jake Maes]

Yeah, sounds pretty light weight.

Quick question for committers: Do we anticipate this affecting our merge
script? Perhaps we won't need the merge script, since we'll be able to use
github directly. Thoughts?

-Jake

On Tue, Jan 15, 2019 at 11:49 AM Prateek Maheshwari 
wrote:

> Thanks for starting the discussion Pawas. I'm +1 (binding) for the
> migration.
>
> - Prateek
>
> On Tue, Jan 15, 2019 at 11:44 AM Pawas Chhokra 
> wrote:
> >
> > Hi all,
> >
> > As mandated by the Apache Infrastructure Team, all git repositories must
> be
> > migrated from git-wip-us.apache.org URL to gitbox.apache.org, as the old
> > service is being decommissioned. This needs to happen before February
> 7th,
> > and this ticket  is to
> > check if migrating Samza on 11 AM, Jan 25, 2019 is acceptable to
> everyone.
> >
> > Thanks & Regards,
> > Pawas Chhokra
>

[ANNOUNCE] New committer announcement

[Xinyu Liu]

Hi, all,

Please join me and the rest of the Samza PMC in welcoming a new committer:

 - Shanthoosh Venkataraman: of, but not limited to, his work on Samza
standalone.

Through his work on developing new features such as host affinity on
standalone and refining existing Samza code base, the PMC trusts Shanthoosh
with the responsibilities of a Samza committer.

Xinyu

Re: [DISCUSS] Mandatory migration of Samza git repo to gitbox.apache.org

[Prateek Maheshwari]

Thanks for starting the discussion Pawas. I'm +1 (binding) for the migration.

- Prateek

On Tue, Jan 15, 2019 at 11:44 AM Pawas Chhokra  wrote:
>
> Hi all,
>
> As mandated by the Apache Infrastructure Team, all git repositories must be
> migrated from git-wip-us.apache.org URL to gitbox.apache.org, as the old
> service is being decommissioned. This needs to happen before February 7th,
> and this ticket  is to
> check if migrating Samza on 11 AM, Jan 25, 2019 is acceptable to everyone.
>
> Thanks & Regards,
> Pawas Chhokra

[DISCUSS] Mandatory migration of Samza git repo to gitbox.apache.org

[Pawas Chhokra]

Hi all,

As mandated by the Apache Infrastructure Team, all git repositories must be
migrated from git-wip-us.apache.org URL to gitbox.apache.org, as the old
service is being decommissioned. This needs to happen before February 7th,
and this ticket  is to
check if migrating Samza on 11 AM, Jan 25, 2019 is acceptable to everyone.

Thanks & Regards,
Pawas Chhokra

[GitHub] samza pull request #868: Minor: Disable outdated sonar-scanner in .travis.ym...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/868


---

[ANNOUNCE] Apache Roadshow Chicago, Call for Presentations

[Trevor Grant]

Hello Devs!


You're receiving this email because you are subscribed to one or more
Apache developer email lists.

I’m writing to let you know about an exciting event coming to the Chicago
area: The Apache Roadshow Chicago.  It will be held May 13th and 14th at
three bars in the Logan Square neighborhood (Revolution Brewing, The
Native, and the Radler).

There will be six tracks:

   -

   Apache in Adtech:  Tell us how Apache works in your advertising stack
   -

   Apache in Fintech: Tell us how Apache works in your finance/insurance
   business
   -

   Apache in Startups: Tell us how you’re using Apache in your startup
   -

   Diversity in Apache: How do we increase and encourage diversity in
   Apache and tech fields overall?
   -

   Made in Chicago: Apache related things made by people in Chicago that
   don’t fall into other buckets
   -

   Project Shark Tank: Do you want more developers or users for your Apache
   project? Come here and pitch it!


This is an exciting chance to learn about how Apache Projects are in use in
production around Chicago, how business users make the decision to use
Apache projects, to learn about exciting new projects that want help from
developers like you, and how/why to increase diversity in tech and IT.

If you have any use cases of Apache products in Adtech, Fintech, or
Startups; if you represent a minority working in tech and have perspectives
to share, if you live in the Chicagoland area and want to highlight some
work you’ve done on an Apache project, or if you want to get other people
excited to come work on your project, then please submit a CFP before the
deadline on February 15th!

Tickets to the Apache Roadshow Chicago are $100; speakers will get a
complimentary ticket.

We’re looking forward to reading your submissions and seeing you there on
May 13-14!

Sincerely,

Trevor Grant

https://www.apachecon.com/chiroadshow19/cfp.html

https://www.apachecon.com/chiroadshow19/register.html