[jira] [Commented] (SAMZA-1631) Do not log message contents on Task callback timeouts

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418291#comment-16418291
 ] 

ASF GitHub Bot commented on SAMZA-1631:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/461


> Do not log message contents on Task callback timeouts
> -
>
> Key: SAMZA-1631
> URL: https://issues.apache.org/jira/browse/SAMZA-1631
> Project: Samza
>  Issue Type: Bug
>Reporter: Prateek Maheshwari
>Assignee: Prateek Maheshwari
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1631) Do not log message contents on Task callback timeouts

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418116#comment-16418116
 ] 

ASF GitHub Bot commented on SAMZA-1631:
---

GitHub user prateekm opened a pull request:

https://github.com/apache/samza/pull/461

SAMZA-1631: Improve logging on Task callback timeout



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/prateekm/samza task-callback-logging

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/461.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #461


commit 44260c4efd159b45823700aa48b61d593e7bc9a3
Author: Prateek Maheshwari 
Date:   2018-03-28T19:19:35Z

Improve logging on Task callback timeout.




> Do not log message contents on Task callback timeouts
> -
>
> Key: SAMZA-1631
> URL: https://issues.apache.org/jira/browse/SAMZA-1631
> Project: Samza
>  Issue Type: Bug
>Reporter: Prateek Maheshwari
>Assignee: Prateek Maheshwari
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1630) Log a thread dump on timeouts

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418108#comment-16418108
 ] 

ASF GitHub Bot commented on SAMZA-1630:
---

GitHub user prateekm opened a pull request:

https://github.com/apache/samza/pull/460

SAMZA-1630: Log a thread dump on timeouts

Would be useful to get a thread dump on timeouts, e.g. for AsyncStreamTask 
callback timeout, container shutdown timeout, heartbeat monitor graceful 
shutdown timeout etc.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/prateekm/samza thread-dump-on-timeout

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/460.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #460


commit 13e423a0613e2582818428974cb8e23533abd120
Author: Prateek Maheshwari 
Date:   2018-03-28T19:20:03Z

Log thread dump on timeouts.




> Log a thread dump on timeouts
> -
>
> Key: SAMZA-1630
> URL: https://issues.apache.org/jira/browse/SAMZA-1630
> Project: Samza
>  Issue Type: Improvement
>Reporter: Prateek Maheshwari
>Priority: Major
>
> Would be useful to get a thread dump on timeouts, e.g. for AsyncStreamTask 
> callback timeout, container shutdown timeout, heartbeat monitor graceful 
> shutdown timeout etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1372) Change Latch Interface to Lock Interface for Samza Standalone with ZK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417934#comment-16417934
 ] 

ASF GitHub Bot commented on SAMZA-1372:
---

Github user PawasChhokra closed the pull request at:

https://github.com/apache/samza/pull/264


> Change Latch Interface to Lock Interface for Samza Standalone with ZK
> -
>
> Key: SAMZA-1372
> URL: https://issues.apache.org/jira/browse/SAMZA-1372
> Project: Samza
>  Issue Type: Improvement
>Reporter: Pawas Chhokra
>Assignee: Pawas Chhokra
>Priority: Major
>
> The current implementation of 'creating intermediate streams' depends on 
> leader election, followed by the leader creating the intermediate streams. We 
> have removed this dependency on leader election and changed the Latch 
> interface to a generic Lock interface. Following this, any processor that 
> acquires the lock can create the intermediate streams, irrespective of 
> whether it is the leader or not. It releases the lock after successful 
> creation of the streams.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1627) Watermark broadcast enhancements

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417798#comment-16417798
 ] 

ASF GitHub Bot commented on SAMZA-1627:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/456


> Watermark broadcast enhancements
> 
>
> Key: SAMZA-1627
> URL: https://issues.apache.org/jira/browse/SAMZA-1627
> Project: Samza
>  Issue Type: Bug
>Reporter: Xinyu Liu
>Assignee: Xinyu Liu
>Priority: Major
>
> Currently each upstream task needs to broadcast to every single partition of 
> intermediate streams in order to aggregate watermarks in the consumers. It's 
> O(n^2). For 256 tasks, 256-partition intermediate stream this can easily 
> result in 64k msg/s if we send watermark every second. To illustrate:
> T1     T2    T3
>  |   \   /\ |  /\ /\ |
> P1      P2    P3
>  
> A better way to do this is to have only one downstream consumer doing the 
> aggregation, and then broadcast to all the partitions. This is safe as we can 
> do a simple proof: if P1 received watermark of t from all T1, T2, and T3, all 
> the messages before t have been published to (P1, P2, P3) already (might not 
> be consumed yet). So P1 can safely broadcast the watermark t to P2 and P3. To 
> illustrate:
> T1     T2     T3
>     \      |       /
>          P1
>        /    \
>       P2  P3
> This reduced the full message count from O(n^2) to O(n). The cost is that 
> this might introduce a few milliseconds delay since we need to exchange the 
> message twice. The benefit clearly wins. In practice, the aggregate consumer 
> can be decided from the (topic.hash() % total partitions) to spread the 
> aggregation if we have multiple intermediate streams.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1293) Enable partition expansion of input streams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16414764#comment-16414764
 ] 

ASF GitHub Bot commented on SAMZA-1293:
---

Github user lindong28 closed the pull request at:

https://github.com/apache/samza/pull/214


> Enable partition expansion of input streams
> ---
>
> Key: SAMZA-1293
> URL: https://issues.apache.org/jira/browse/SAMZA-1293
> Project: Samza
>  Issue Type: New Feature
>Reporter: Dong Lin
>Priority: Major
>
> Right now, Samza does not allow partitions of the input stream to increase 
> after a stateful job is created. This causes problem when Kafka is used as 
> the input system, because we need to expand partitions of an existing topic 
> as the byte-in-rate of the topic increases over time in order to limit the 
> size of the maximum partition in Kafka. Kafka broker may have performance 
> issue if the size of a given partition is too large.
> This patch provides a solution to increase partition number of the input 
> streams of a stateful Samza job while still ensuring the correctness of Samze 
> job output. The solution should work when Kafka is used as the input system. 
> We expect this solution to work similarly with other input system as well. 
> The motivation of increasing partition number of Kafka topic is 1) increase 
> performance of Kafka broker and 2) increase throughput of Kafka consumer in 
> the Samza container.
> See SEP-5 
> (https://cwiki.apache.org/confluence/display/SAMZA/SEP-5%3A+Enable+partition+expansion+of+input+streams)
>  for the design and the interface change of this patch.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1571) Make Eventhubs system configs compatible with Samza standalone.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410603#comment-16410603
 ] 

ASF GitHub Bot commented on SAMZA-1571:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/453


> Make Eventhubs system configs compatible with Samza standalone.
> ---
>
> Key: SAMZA-1571
> URL: https://issues.apache.org/jira/browse/SAMZA-1571
> Project: Samza
>  Issue Type: Bug
>Reporter: Boris Shkolnik
>Assignee: Jagadish
>Priority: Major
>
> EventHubs configs are using the old style configs (system.*.streams.*.config).
> It needs to be moved to the new stream-id configs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1623) AvroDataFileHdfsWriter should include avro as the file suffix

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408507#comment-16408507
 ] 

ASF GitHub Bot commented on SAMZA-1623:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/452


> AvroDataFileHdfsWriter should include avro as the file suffix
> -
>
> Key: SAMZA-1623
> URL: https://issues.apache.org/jira/browse/SAMZA-1623
> Project: Samza
>  Issue Type: Task
>Reporter: Hai
>Assignee: Hai
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1623) AvroDataFileHdfsWriter should include avro as the file suffix

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408339#comment-16408339
 ] 

ASF GitHub Bot commented on SAMZA-1623:
---

GitHub user lhaiesp opened a pull request:

https://github.com/apache/samza/pull/452

SAMZA-1623: include avro as the file suffix for hdfs producer

AvroDataFileHdfsWriter should include avro as the file suffix as some pig 
jobs couldn't read the avro files if they don't come with the proper suffix

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lhaiesp/samza master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/452.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #452


commit 527b49cbd0553ffbcba2a5b7459ba77ccbd1d1c2
Author: Hai Lu 
Date:   2018-03-21T18:00:01Z

SAMZA-1623: AvroDataFileHdfsWriter should include avro as the file suffix




> AvroDataFileHdfsWriter should include avro as the file suffix
> -
>
> Key: SAMZA-1623
> URL: https://issues.apache.org/jira/browse/SAMZA-1623
> Project: Samza
>  Issue Type: Task
>Reporter: Hai
>Assignee: Hai
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1608) Add hidden config to enable explicit stream creation in StreamAppender due to bug.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402433#comment-16402433
 ] 

ASF GitHub Bot commented on SAMZA-1608:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/442


> Add hidden config to enable explicit stream creation in StreamAppender due to 
> bug.
> --
>
> Key: SAMZA-1608
> URL: https://issues.apache.org/jira/browse/SAMZA-1608
> Project: Samza
>  Issue Type: Bug
>Reporter: Daniel Nishimura
>Assignee: Daniel Nishimura
>Priority: Major
>
> Explicitly creating the stream in StreamAppender intermittently causes a hang.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1622) avro writer in hdfs system producer to support generic record

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402124#comment-16402124
 ] 

ASF GitHub Bot commented on SAMZA-1622:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/449


> avro writer in hdfs system producer to support generic record
> -
>
> Key: SAMZA-1622
> URL: https://issues.apache.org/jira/browse/SAMZA-1622
> Project: Samza
>  Issue Type: Bug
>Reporter: Hai
>Assignee: Hai
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1615) Fix a couple of issues in ControlMessageSender

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401277#comment-16401277
 ] 

ASF GitHub Bot commented on SAMZA-1615:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/444


> Fix a couple of issues in ControlMessageSender 
> ---
>
> Key: SAMZA-1615
> URL: https://issues.apache.org/jira/browse/SAMZA-1615
> Project: Samza
>  Issue Type: Bug
>Reporter: Xinyu Liu
>Assignee: Xinyu Liu
>Priority: Major
>
> Two issues I found during testing:
> 1) medaDataCache.getSystemStreamMetadata(): if we pass in partitionOnly to be 
> true, it will actually not store the metadata into the cache, resulting every 
> call being another query to kafka. I turned off the partitionOnly in order to 
> make it in the cache.
> 2) change the log for info to debug.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1622) avro writer in hdfs system producer to support generic record

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400905#comment-16400905
 ] 

ASF GitHub Bot commented on SAMZA-1622:
---

GitHub user lhaiesp opened a pull request:

https://github.com/apache/samza/pull/449

SAMZA-1622: avro writer to support generic record

avro writer in HDFS system producer to support generic record

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lhaiesp/samza master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/449.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #449






> avro writer in hdfs system producer to support generic record
> -
>
> Key: SAMZA-1622
> URL: https://issues.apache.org/jira/browse/SAMZA-1622
> Project: Samza
>  Issue Type: Bug
>Reporter: Hai
>Assignee: Hai
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1564) StreamAppender needs to explicitly have a call to SystemAdmin.createStream()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399417#comment-16399417
 ] 

ASF GitHub Bot commented on SAMZA-1564:
---

GitHub user shanthoosh opened a pull request:

https://github.com/apache/samza/pull/448

SAMZA-1564: Host affinity in standalone.

SEP-11 Patch 1:  Introducing MetadataStore interface and the associated 
classes.

Followup patches will contain the TaskNameGrouper changes and other changes 
that are part of SEP-11.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shanthoosh/samza introduce_meta_store

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/448.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #448


commit 7e0401bb6f97107763990790c66ba0dc9aaa
Author: Shanthoosh Venkataraman 
Date:   2018-03-14T16:10:45Z

SEP-11 Part 1:

Introduce Metadata store interface and the associated classes.




> StreamAppender needs to explicitly have a call to SystemAdmin.createStream()
> 
>
> Key: SAMZA-1564
> URL: https://issues.apache.org/jira/browse/SAMZA-1564
> Project: Samza
>  Issue Type: Bug
>Reporter: Daniel Nishimura
>Assignee: Daniel Nishimura
>Priority: Major
>
> StreamAppender currently relies on auto stream creation that may not work on 
> the configured system. For example, if the stream is a Kafka stream, auto 
> topic creation could be turned off.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1611) BootstrappingChooser should use systemAdmin offsetComparator API to compare the offsets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397920#comment-16397920
 ] 

ASF GitHub Bot commented on SAMZA-1611:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/443


> BootstrappingChooser should use systemAdmin offsetComparator API to compare 
> the offsets
> ---
>
> Key: SAMZA-1611
> URL: https://issues.apache.org/jira/browse/SAMZA-1611
> Project: Samza
>  Issue Type: Bug
>Reporter: Aditya
>Priority: Major
>
> BootstrappingChooser currently uses string equality check with the NEWEST or 
> UPCOMING offset to determine the end of bootstrap rather than using 
> systemAdmin offsetComparator API. This string equality check might not work 
> for all systems.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1618) fix HdfsFileSystemAdapter to get files recursively

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397510#comment-16397510
 ] 

ASF GitHub Bot commented on SAMZA-1618:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/447


> fix HdfsFileSystemAdapter to get files recursively
> --
>
> Key: SAMZA-1618
> URL: https://issues.apache.org/jira/browse/SAMZA-1618
> Project: Samza
>  Issue Type: Bug
>Reporter: Hai
>Assignee: Hai
>Priority: Major
>
> Right now HdfsFileSystemAdapter doesn't handle subfolder



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1609) Refactor SamzaContainer to take in SystemFactories

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395512#comment-16395512
 ] 

ASF GitHub Bot commented on SAMZA-1609:
---

GitHub user Sanil15 opened a pull request:

https://github.com/apache/samza/pull/445

SAMZA-1609 Refactoring SamzaContainer to take SystemFactories as an 
argument 

- This is a pre-req to SEP-8 
(https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=71013043)
- A subtask to https://issues.apache.org/jira/browse/SAMZA-1395
- I have updated tests/dependencies to the best of my knowledge, please 
advice if some refactoring can be improved

Tested with ./gradlew clean build 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Sanil15/samza SAMZA-1395

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/445.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #445


commit 683c853ddc868f3375519629988df546b0432416
Author: sanil15 
Date:   2018-03-10T00:43:01Z

Refactoring SamzaContainer to take SystemFactories as an argument




> Refactor SamzaContainer to take in SystemFactories
> --
>
> Key: SAMZA-1609
> URL: https://issues.apache.org/jira/browse/SAMZA-1609
> Project: Samza
>  Issue Type: Sub-task
>Reporter: Bharath Kumarasubramanian
>Assignee: Sanil Jain
>Priority: Major
>
> Currently, SamzaContainer is responsible for creating SystemFactories using 
> class loading. We want to move this to runner since it is responsible for 
> creating intermediate streams and seems a natural resting place for factory 
> creation. 
> Additionally, this is a pre-requisite for in-memory system to initialize the 
> data source.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1615) Fix a couple of issues in ControlMessageSender

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395487#comment-16395487
 ] 

ASF GitHub Bot commented on SAMZA-1615:
---

GitHub user xinyuiscool opened a pull request:

https://github.com/apache/samza/pull/444

SAMZA-1615: Fix a couple of issues in ControlMessageSender

Two issues I found during testing: 1) 
medaDataCache.getSystemStreamMetadata(): if we pass in partitionOnly to be 
true, it will actually not store the metadata into the cache, resulting every 
call being another query to kafka. I turned off the partitionOnly in order to 
make it in the cache. 2) change the log for info to debug.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xinyuiscool/samza SAMZA-1615

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/444.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #444


commit c92a2c66458545ed6903ca75ab6751bc6a85cd8d
Author: xinyuiscool 
Date:   2018-03-12T16:41:14Z

SAMZA-1615: Fix a couple of issues in ControlMessageSender




> Fix a couple of issues in ControlMessageSender 
> ---
>
> Key: SAMZA-1615
> URL: https://issues.apache.org/jira/browse/SAMZA-1615
> Project: Samza
>  Issue Type: Bug
>Reporter: Xinyu Liu
>Assignee: Xinyu Liu
>Priority: Major
>
> Two issues I found during testing:
> 1) medaDataCache.getSystemStreamMetadata(): if we pass in partitionOnly to be 
> true, it will actually not store the metadata into the cache, resulting every 
> call being another query to kafka. I turned off the partitionOnly in order to 
> make it in the cache.
> 2) change the log for info to debug.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1608) Add hidden config to enable explicit stream creation in StreamAppender due to bug.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393176#comment-16393176
 ] 

ASF GitHub Bot commented on SAMZA-1608:
---

GitHub user dnishimura opened a pull request:

https://github.com/apache/samza/pull/442

SAMZA-1608 : Add hidden config to enable explicit stream creation in 
StreamAppender due to bug.

Due to a intermittent bug that causes the explicit stream creation in 
`StreamAppender` to hang, a hidden config is added to enable/disable explicit 
stream creation. By default this is disabled, which reverts to the previous 
behavior.

When the intermittent hang bug is fixed, the config will either be removed 
or made public.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dnishimura/samza 
samza-1608-disable-streamappender-create-stream

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/442.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #442


commit e56f81c4222d17d32bc09c337c6cafd901e98247
Author: Daniel Nishimura 
Date:   2018-03-09T16:59:43Z

Add hidden config to enable explicit stream creation in StreamAppender.




> Add hidden config to enable explicit stream creation in StreamAppender due to 
> bug.
> --
>
> Key: SAMZA-1608
> URL: https://issues.apache.org/jira/browse/SAMZA-1608
> Project: Samza
>  Issue Type: Bug
>Reporter: Daniel Nishimura
>Assignee: Daniel Nishimura
>Priority: Major
>
> Explicitly creating the stream in StreamAppender intermittently causes a hang.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1589) Reduce failure retry duration in KafkaCheckpointManager.writeCheckpoint.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392259#comment-16392259
 ] 

ASF GitHub Bot commented on SAMZA-1589:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/438


> Reduce failure retry duration in KafkaCheckpointManager.writeCheckpoint.
> 
>
> Key: SAMZA-1589
> URL: https://issues.apache.org/jira/browse/SAMZA-1589
> Project: Samza
>  Issue Type: Improvement
>Reporter: Shanthoosh Venkataraman
>Assignee: Shanthoosh Venkataraman
>Priority: Major
>
> In the existing implementation of KafkaCheckpointManager.writecheckpoint, 
> failure retry duration is set to 1.5 hours.
> In some scenarios, this larger retry time will delay the irrecoverable 
> failure propagation to the clients and is undesirable.
> Suggested proposal is to reduce the failure retry time to 15 minutes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1584) LocalApplicationRunner.waitForFinish blocks indefinitely.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392076#comment-16392076
 ] 

ASF GitHub Bot commented on SAMZA-1584:
---

GitHub user shanthoosh opened a pull request:

https://github.com/apache/samza/pull/441

SAMZA-1584: LocalApplicationRunner.waitForFinish blocks indefinitely.

Changes. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shanthoosh/samza SAMZA-1584

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/441.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #441


commit 6ee5195f96b6cd5e4569c71257a810811ec4fc35
Author: Shanthoosh Venkataraman 
Date:   2018-03-07T21:49:32Z

SAMZA-1584: LocalApplicationRunner.waitForFinish blocks indefinitely.




> LocalApplicationRunner.waitForFinish blocks indefinitely.
> -
>
> Key: SAMZA-1584
> URL: https://issues.apache.org/jira/browse/SAMZA-1584
> Project: Samza
>  Issue Type: Bug
>Reporter: Shanthoosh Venkataraman
>Assignee: Shanthoosh Venkataraman
>Priority: Major
>
> Recommended solution to kill streamApplication in standalone is the following:
> {code:java}
> LocalApplicationRunner.kill(streamApp);
> LocalApplicationRunner.waitForFinish();{code}
> `LocalApplicationRunner.kill(streamApplication)` invokes 
> `StreamProcessor.stop()` and  `StreamProcessor.stop()` invokes 
> `SamzaContainer.shutdown()` to kill the samza-container. 
> `SamzaContainer.shutdown()` marks the shutdown flag of `AsyncRunLoop` and 
> shutdown flag will be taken into account by `AsyncRunLoop` when it chooses 
> the next message to route to tasks. 
> Actual shutdown sequence of StreamProcessor and ZkJobCoordinator is triggered 
> on SamzaContainerListener.onContainerStop() callback which is fired when the 
> SamzaContainer is completely shutdown. LocalApplication.awaitForFinish() will 
> block forever in following two scenarios:
>  * If any of the AsyncRunLoop operation(commit, window, process) is going on 
> indefinitely, then the marked shutdown flag will not be seen by the 
> AsyncRunLoop.
>  * If any step in the SamzaContainer shutdown sequence blocks indefinitely.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1607) SAMZA-1607: Handle ZkNoNodeExistsException in zkUtils.readProcessorData

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390263#comment-16390263
 ] 

ASF GitHub Bot commented on SAMZA-1607:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/437


> SAMZA-1607: Handle ZkNoNodeExistsException in zkUtils.readProcessorData
> ---
>
> Key: SAMZA-1607
> URL: https://issues.apache.org/jira/browse/SAMZA-1607
> Project: Samza
>  Issue Type: Bug
>Reporter: Shanthoosh Venkataraman
>Assignee: Shanthoosh Venkataraman
>Priority: Major
>
> Existing implementation of reading the data of ephemeral processor nodes in 
> zookeeper happens in two steps.
>    A. Fetch the list of ephemeral processor nodes.
>    B. Read the data of each processor node from the list. 
> A ephemeral zookeeper node present in step A might be unavailable in the step 
> B. This exception in unhandled currently and can kill the leader processor 
> unnecessarily. Here's the related exception observed in a dev setup.
> {code:java}
> org.apache.samza.SamzaException: Cannot read ZK node: 
> /app-test-app-name-9fba7675-36e3-4a6e-8934-4cad6a8ebab0-test-app-id-9fba7675-36e3-4a6e-8934-4cad6a8ebab0/test-app-name-9fba7675-36e3-4a6e-8934-4cad6a8ebab0-test-app-id-9fba7675-36e3-4a6e-8934-4cad6a8ebab0-coordinationData/processors/01
> at org.apache.samza.zk.ZkUtils.readProcessorData(ZkUtils.java:232)
> at org.apache.samza.zk.ZkUtils.getActiveProcessorsIDs(ZkUtils.java:255)
> at 
> org.apache.samza.zk.ZkJobCoordinator.getActualProcessorIds(ZkJobCoordinator.java:292)
> at 
> org.apache.samza.zk.ZkJobCoordinator.doOnProcessorChange(ZkJobCoordinator.java:194)
> at 
> org.apache.samza.zk.ZkJobCoordinator.lambda$onProcessorChange$1(ZkJobCoordinator.java:188)
> at 
> org.apache.samza.zk.ScheduleAfterDebounceTime.lambda$getScheduleableAction$0(ScheduleAfterDebounceTime.java:134)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.I0Itec.zkclient.exception.ZkNoNodeException: 
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = 
> NoNode for 
> /app-test-app-name-9fba7675-36e3-4a6e-8934-4cad6a8ebab0-test-app-id-9fba7675-36e3-4a6e-8934-4cad6a8ebab0/test-app-name-9fba7675-36e3-4a6e-8934-4cad6a8ebab0-test-app-id-9fba7675-36e3-4a6e-8934-4cad6a8ebab0-coordinationData/processors/01
> at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
> at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:1001)
> at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:1100)
> at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:1095)
> at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:1084)
> at org.apache.samza.zk.ZkUtils.readProcessorData(ZkUtils.java:226)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1602) Samza-Test: Moving StreamAssert from src/test to src/main

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390257#comment-16390257
 ] 

ASF GitHub Bot commented on SAMZA-1602:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/439


> Samza-Test: Moving StreamAssert from src/test to src/main
> -
>
> Key: SAMZA-1602
> URL: https://issues.apache.org/jira/browse/SAMZA-1602
> Project: Samza
>  Issue Type: Task
>Reporter: Sanil Jain
>Assignee: Sanil Jain
>Priority: Minor
>
> * StreamAssert.java is currently placed under 
> samza-test/src/test/java/org.apache.samza.test.util, but it is actually not a 
> util file and the location is not intuitive
> * Moving StreamAssert.java to 
> samza-test/src/MAIN/java/org.apache.samza.test.framework
> * Updating relevant gradle configuration to make this change viable 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1602) Samza-Test: Moving StreamAssert from src/test to src/main

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389931#comment-16389931
 ] 

ASF GitHub Bot commented on SAMZA-1602:
---

GitHub user Sanil15 opened a pull request:

https://github.com/apache/samza/pull/439

SAMZA-1602 : Moving class StreamAssert from src/test to src/main



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Sanil15/samza SAMZA-1602

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/439.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #439


commit 5687acabdc2009456dc69f0ef451bb9ec53564c4
Author: sanil15 
Date:   2018-03-07T18:23:37Z

SAMZA-1602 Moving class StreamAssert from src/test to src/main




> Samza-Test: Moving StreamAssert from src/test to src/main
> -
>
> Key: SAMZA-1602
> URL: https://issues.apache.org/jira/browse/SAMZA-1602
> Project: Samza
>  Issue Type: Task
>Reporter: Sanil Jain
>Assignee: Sanil Jain
>Priority: Minor
>
> * StreamAssert.java is currently placed under 
> samza-test/src/test/java/org.apache.samza.test.util, but it is actually not a 
> util file and the location is not intuitive
> * Moving StreamAssert.java to 
> samza-test/src/MAIN/java/org.apache.samza.test.framework
> * Updating relevant gradle configuration to make this change viable 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1589) Reduce failure retry duration in KafkaCheckpointManager.writeCheckpoint.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389317#comment-16389317
 ] 

ASF GitHub Bot commented on SAMZA-1589:
---

GitHub user shanthoosh opened a pull request:

https://github.com/apache/samza/pull/438

SAMZA-1589: Reduce failure retry duration in 
KafkaCheckpointManager.writeCheckpoint



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shanthoosh/samza SAMZA-1589

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/438.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #438


commit c6a127d7d2eaa5c81a33bc885c04e0e23e818c25
Author: Shanthoosh Venkataraman 
Date:   2018-03-07T09:25:33Z

SAMZA-1589: Reduce failure retry duration in 
KafkaCheckpointManager.writeCheckpoint




> Reduce failure retry duration in KafkaCheckpointManager.writeCheckpoint.
> 
>
> Key: SAMZA-1589
> URL: https://issues.apache.org/jira/browse/SAMZA-1589
> Project: Samza
>  Issue Type: Improvement
>Reporter: Shanthoosh Venkataraman
>Assignee: Shanthoosh Venkataraman
>Priority: Major
>
> In the existing implementation of KafkaCheckpointManager.writecheckpoint, 
> failure retry duration is set to 1.5 hours.
> In some scenarios, this larger retry time will delay the irrecoverable 
> failure propagation to the clients and is undesirable.
> Suggested proposal is to reduce the failure retry time to 15 minutes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1607) Handle ZkNodeNotExists exception in zkUtils.readProcessorData

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389275#comment-16389275
 ] 

ASF GitHub Bot commented on SAMZA-1607:
---

GitHub user shanthoosh opened a pull request:

https://github.com/apache/samza/pull/437

SAMZA-1607: Handle ZkNodeNotExistsException in zkUtils.readProcessorData



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shanthoosh/samza 
fix_zkutils_get_processor_data

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/437.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #437


commit 2c6d5f9cee4d833d8f63823ee4078f41a726203f
Author: Shanthoosh Venkataraman 
Date:   2018-02-12T23:25:36Z

SAMZA-1607: Handle ZkNodeNotExists exception in zkUtils.readProcessorData().




> Handle ZkNodeNotExists exception in zkUtils.readProcessorData
> -
>
> Key: SAMZA-1607
> URL: https://issues.apache.org/jira/browse/SAMZA-1607
> Project: Samza
>  Issue Type: Bug
>Reporter: Shanthoosh Venkataraman
>Assignee: Shanthoosh Venkataraman
>Priority: Major
>
> Existing implementation of reading the data of ephemeral processor nodes in 
> zookeeper happens in two steps.
>    A. Fetch the list of ephemeral processor nodes.
>    B. Read the data of each processor node from the list. 
> A ephemeral zookeeper node present in step A might be unavailable in the step 
> B. This exception in unhandled currently and can kill the leader processor 
> unnecessarily. Here's the related exception observed in a dev setup.
> {code:java}
> org.apache.samza.SamzaException: Cannot read ZK node: 
> /app-test-app-name-9fba7675-36e3-4a6e-8934-4cad6a8ebab0-test-app-id-9fba7675-36e3-4a6e-8934-4cad6a8ebab0/test-app-name-9fba7675-36e3-4a6e-8934-4cad6a8ebab0-test-app-id-9fba7675-36e3-4a6e-8934-4cad6a8ebab0-coordinationData/processors/01
> at org.apache.samza.zk.ZkUtils.readProcessorData(ZkUtils.java:232)
> at org.apache.samza.zk.ZkUtils.getActiveProcessorsIDs(ZkUtils.java:255)
> at 
> org.apache.samza.zk.ZkJobCoordinator.getActualProcessorIds(ZkJobCoordinator.java:292)
> at 
> org.apache.samza.zk.ZkJobCoordinator.doOnProcessorChange(ZkJobCoordinator.java:194)
> at 
> org.apache.samza.zk.ZkJobCoordinator.lambda$onProcessorChange$1(ZkJobCoordinator.java:188)
> at 
> org.apache.samza.zk.ScheduleAfterDebounceTime.lambda$getScheduleableAction$0(ScheduleAfterDebounceTime.java:134)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.I0Itec.zkclient.exception.ZkNoNodeException: 
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = 
> NoNode for 
> /app-test-app-name-9fba7675-36e3-4a6e-8934-4cad6a8ebab0-test-app-id-9fba7675-36e3-4a6e-8934-4cad6a8ebab0/test-app-name-9fba7675-36e3-4a6e-8934-4cad6a8ebab0-test-app-id-9fba7675-36e3-4a6e-8934-4cad6a8ebab0-coordinationData/processors/01
> at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
> at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:1001)
> at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:1100)
> at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:1095)
> at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:1084)
> at org.apache.samza.zk.ZkUtils.readProcessorData(ZkUtils.java:226)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1498) Support arbitrary system clock timer in operators

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388908#comment-16388908
 ] 

ASF GitHub Bot commented on SAMZA-1498:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/419


> Support arbitrary system clock timer in operators
> -
>
> Key: SAMZA-1498
> URL: https://issues.apache.org/jira/browse/SAMZA-1498
> Project: Samza
>  Issue Type: Bug
>Reporter: Prateek Maheshwari
>Assignee: Xinyu Liu
>Priority: Minor
> Fix For: 0.15.0
>
>
> We need the capability of supporting any system-clock timers for user code. 
> So user can register any timer as they like and they will get a callback in 
> their functions. Internally we will be able to register the timer inside a 
> scheduler and AsyncRunLoop will query the timers and fire the callback 
> accordingly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1600) Changelog topics for batch jobs should not use combination of delete+compact cleanup policy in Kafka

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387222#comment-16387222
 ] 

ASF GitHub Bot commented on SAMZA-1600:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/435


> Changelog topics for batch jobs should not use combination of delete+compact 
> cleanup policy in Kafka
> 
>
> Key: SAMZA-1600
> URL: https://issues.apache.org/jira/browse/SAMZA-1600
> Project: Samza
>  Issue Type: Bug
>Reporter: Yi Pan (Data Infrastructure)
>Assignee: Yi Pan (Data Infrastructure)
>Priority: Major
>
> We have encountered a bug in Kafka 0.11 that when a topic is configured to 
> use a combination of cleanup policy delete+compact, the broker enters a 
> deadlock situation.
> Currently, we set the delete+compact policy for changelog topics in batch 
> jobs to reduce the storage requirement and the additional GC effort required. 
> However, before the above mentioned Kafka bug is fixed, we will disable this 
> feature for now.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1600) Changelog topics for batch jobs should not use combination of delete+compact cleanup policy in Kafka

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387095#comment-16387095
 ] 

ASF GitHub Bot commented on SAMZA-1600:
---

GitHub user nickpan47 opened a pull request:

https://github.com/apache/samza/pull/435

SAMZA-1600: remove the combination of cleanup policy "compact,delete"…

… in changelog topic properties

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickpan47/samza SAMZA-1600

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/435.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #435


commit 6549c4f3fe8cb3ca54f60a6f6f966a3c24343fde
Author: Yi Pan (Data Infrastructure) 
Date:   2018-03-06T00:29:19Z

SAMZA-1600: remove the combination of cleanup policy "compact,delete" in 
changelog topic properties




> Changelog topics for batch jobs should not use combination of delete+compact 
> cleanup policy in Kafka
> 
>
> Key: SAMZA-1600
> URL: https://issues.apache.org/jira/browse/SAMZA-1600
> Project: Samza
>  Issue Type: Bug
>Reporter: Yi Pan (Data Infrastructure)
>Assignee: Yi Pan (Data Infrastructure)
>Priority: Major
>
> We have encountered a bug in Kafka 0.11 that when a topic is configured to 
> use a combination of cleanup policy delete+compact, the broker enters a 
> deadlock situation.
> Currently, we set the delete+compact policy for changelog topics in batch 
> jobs to reduce the storage requirement and the additional GC effort required. 
> However, before the above mentioned Kafka bug is fixed, we will disable this 
> feature for now.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1460) StreamAppender does not explicitly create logging topic

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387005#comment-16387005
 ] 

ASF GitHub Bot commented on SAMZA-1460:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/423


> StreamAppender does not explicitly create logging topic
> ---
>
> Key: SAMZA-1460
> URL: https://issues.apache.org/jira/browse/SAMZA-1460
> Project: Samza
>  Issue Type: Bug
>Reporter: Yi Pan (Data Infrastructure)
>Assignee: Daniel Nishimura
>Priority: Major
>
> In StreamAppender, the logging topic is not explicitly created. We will be 
> relying on Kafka cluster topic auto-creation feature to create the logging 
> topic:
> {code}
>   protected void setupSystem() {
> config = getConfig();
> SystemFactory systemFactory = null;
> Log4jSystemConfig log4jSystemConfig = new Log4jSystemConfig(config);
> if (streamName == null) {
>   streamName = getStreamName(log4jSystemConfig.getJobName(), 
> log4jSystemConfig.getJobId());
> }
> String systemName = log4jSystemConfig.getSystemName();
> String systemFactoryName = log4jSystemConfig.getSystemFactory(systemName);
> if (systemFactoryName != null) {
>   systemFactory = Util.getObj(systemFactoryName);
> } else {
>   throw new SamzaException("Could not figure out \"" + systemName + "\" 
> system factory for log4j StreamAppender to use");
> }
> setSerde(log4jSystemConfig, systemName, streamName);
> systemProducer = systemFactory.getProducer(systemName, config, new 
> MetricsRegistryMap());
> systemStream = new SystemStream(systemName, streamName);
> systemProducer.register(SOURCE);
> systemProducer.start();
> log.info(SOURCE + " has been registered in " + systemName + ". So all the 
> logs will be sent to " + streamName
> + " in " + systemName + ". Logs are partitioned by " + key);
>   }
> {code}
> It would be better to explicitly create the logging topic via 
> SystemAdmin.createStream() to work w/ logging systems that do not support 
> topic auto-creation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1598) Cleanup README.md file

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384922#comment-16384922
 ] 

ASF GitHub Bot commented on SAMZA-1598:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/434


> Cleanup README.md file
> --
>
> Key: SAMZA-1598
> URL: https://issues.apache.org/jira/browse/SAMZA-1598
> Project: Samza
>  Issue Type: Bug
>Reporter: Daniel Nishimura
>Assignee: Daniel Nishimura
>Priority: Minor
>
> Update links pointing to old documentation. Remove Jenkins build badge, in 
> favor of Travis-CI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1598) Cleanup README.md file

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383194#comment-16383194
 ] 

ASF GitHub Bot commented on SAMZA-1598:
---

GitHub user dnishimura opened a pull request:

https://github.com/apache/samza/pull/434

SAMZA-1598 : Cleanup README.md file

Update old links. Remove Jenkins build badge, in favor of Travis-CI.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dnishimura/samza samza-1598-clean-readme

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/434.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #434


commit 75335ad3fd1a1aecb1a0b98b0d45f513aa69a59e
Author: Daniel Nishimura 
Date:   2018-03-02T05:29:07Z

Update old links. Remove Jenkins build badge, in favor of Travis-CI.




> Cleanup README.md file
> --
>
> Key: SAMZA-1598
> URL: https://issues.apache.org/jira/browse/SAMZA-1598
> Project: Samza
>  Issue Type: Bug
>Reporter: Daniel Nishimura
>Assignee: Daniel Nishimura
>Priority: Minor
>
> Update links pointing to old documentation. Remove Jenkins build badge, in 
> favor of Travis-CI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1596) Staging direcroty in HDFS config

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382461#comment-16382461
 ] 

ASF GitHub Bot commented on SAMZA-1596:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/431


> Staging direcroty in HDFS config
> 
>
> Key: SAMZA-1596
> URL: https://issues.apache.org/jira/browse/SAMZA-1596
> Project: Samza
>  Issue Type: Bug
>Affects Versions: 0.14.0, 0.15.0
>Reporter: Akim Akimov
>Priority: Major
> Fix For: 0.15.0
>
>
> When we instantiate a HDFS config staging directory we missing a formatter 
> for getStagingDirectory so systems.hdfs-system-name.stagingDirectory does not 
> parse from config, but only systems.%s.stagingDirectory only parses instead.
>  
> Solution is to add formatter to getStagingDirectory method.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1595) Fix scalacCompileOptions format to build with zinc scala compiler.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373740#comment-16373740
 ] 

ASF GitHub Bot commented on SAMZA-1595:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/430


> Fix scalacCompileOptions format to build with zinc scala compiler.
> --
>
> Key: SAMZA-1595
> URL: https://issues.apache.org/jira/browse/SAMZA-1595
> Project: Samza
>  Issue Type: Improvement
>Reporter: Shanthoosh Venkataraman
>Assignee: Shanthoosh Venkataraman
>Priority: Major
>
> Zinc scala compiler(part of gradle version >= 3.0) expects the scala 
> compilation arguments as a list(where each compilation argument is an element 
> of the list).
> In samza, the compilation arguments are concatenated into a single string and 
> passed to the compiler.
> This causes build failures when samza is built with gradle version > 3.0.
> Existing ant scala compiler used to build samza in open source accepts the 
> compilation arguments  as both list and string.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1595) Fix scalacCompileOptions format to build with zinc scala compiler.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373716#comment-16373716
 ] 

ASF GitHub Bot commented on SAMZA-1595:
---

GitHub user shanthoosh opened a pull request:

https://github.com/apache/samza/pull/430

SAMZA-1595: Fix scalacCompileOptions format to build with zinc scala 
compiler.

Zinc scala compiler(part of gradle version >= 3.0) expects the scala 
compilation arguments as a list(where each compilation argument is an element 
of the list).

In samza, the compilation arguments are concatenated into a single string 
and passed to the compiler.

This causes build failures when samza is built with Zinc scala compiler.

Existing ant scala compiler used to build samza in open source accepts the 
compilation arguments both as list and string.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shanthoosh/samza master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/430.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #430


commit a1ff8325a820a082c93eb552e3dcd21f45306d62
Author: Shanthoosh Venkataraman 
Date:   2018-02-22T23:42:33Z

Fix scalacCompileOptions format to build with zinc scala compiler.




> Fix scalacCompileOptions format to build with zinc scala compiler.
> --
>
> Key: SAMZA-1595
> URL: https://issues.apache.org/jira/browse/SAMZA-1595
> Project: Samza
>  Issue Type: Improvement
>Reporter: Shanthoosh Venkataraman
>Assignee: Shanthoosh Venkataraman
>Priority: Major
>
> Zinc scala compiler(part of gradle version >= 3.0) expects the scala 
> compilation arguments as a list(where each compilation argument is an element 
> of the list).
> In samza, the compilation arguments are concatenated into a single string and 
> passed to the compiler.
> This causes build failures when samza is built with gradle version > 3.0.
> Existing ant scala compiler used to build samza in open source accepts the 
> compilation arguments  as both list and string.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1594) Fix ScalaCompileOptions errors originating from build.gradle.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373708#comment-16373708
 ] 

ASF GitHub Bot commented on SAMZA-1594:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/427


> Fix ScalaCompileOptions errors originating from build.gradle.
> -
>
> Key: SAMZA-1594
> URL: https://issues.apache.org/jira/browse/SAMZA-1594
> Project: Samza
>  Issue Type: Improvement
>Reporter: Shanthoosh Venkataraman
>Assignee: Shanthoosh Venkataraman
>Priority: Major
>
> When samza repository is built with the gradle version greater than 3.0, we 
> notice the following build failure.
> {code:java}
> No such property: useAnt for class: 
> org.gradle.api.tasks.scala.ScalaCompileOptions  
> {code}
> This needs to be fixed to build samza with  gradle version >= 3.0.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1594) Fix ScalaCompileOptions errors originating from build.gradle.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373410#comment-16373410
 ] 

ASF GitHub Bot commented on SAMZA-1594:
---

GitHub user shanthoosh opened a pull request:

https://github.com/apache/samza/pull/427

SAMZA-1594: Fix ScalaCompileOptions errors with higher gradle version.

When samza repository is built with the gradle version greater than 3.0, we 
notice the following build failure.

No such property: useAnt for class: 
org.gradle.api.tasks.scala.ScalaCompileOptions  

This needs to be fixed to build samza with  gradle version >= 3.0.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shanthoosh/samza fix_scalac_compile_options

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/427.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #427


commit 411f1c407607797e6b8e1fdce5aa6af1f9534469
Author: Shanthoosh Venkataraman 
Date:   2018-02-22T20:37:44Z

SAMZA-1594: Fix ScalaCompileOptions errors with higher gradle version.




> Fix ScalaCompileOptions errors originating from build.gradle.
> -
>
> Key: SAMZA-1594
> URL: https://issues.apache.org/jira/browse/SAMZA-1594
> Project: Samza
>  Issue Type: Improvement
>Reporter: Shanthoosh Venkataraman
>Assignee: Shanthoosh Venkataraman
>Priority: Major
>
> When samza repository is built with the gradle version greater than 3.0, we 
> notice the following build failure.
> {code:java}
> No such property: useAnt for class: 
> org.gradle.api.tasks.scala.ScalaCompileOptions  
> {code}
> This needs to be fixed to build samza with  gradle version >= 3.0.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1593) Upgrade gradle nexus plugin.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373408#comment-16373408
 ] 

ASF GitHub Bot commented on SAMZA-1593:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/426


> Upgrade gradle nexus plugin.
> 
>
> Key: SAMZA-1593
> URL: https://issues.apache.org/jira/browse/SAMZA-1593
> Project: Samza
>  Issue Type: Improvement
>Reporter: Shanthoosh Venkataraman
>Assignee: Shanthoosh Venkataraman
>Priority: Major
>
> Current nexus plugin which samza repository is using is not compatible with 
> the gradle versions greater than 3.0.
> For doing the ligradle jvm migration at linkedin, we need to update the nexus 
> gradle plugin to the latest version. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1593) Upgrade gradle nexus plugin.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373372#comment-16373372
 ] 

ASF GitHub Bot commented on SAMZA-1593:
---

GitHub user shanthoosh opened a pull request:

https://github.com/apache/samza/pull/426

SAMZA-1593: Upgrade gradle nexus plugin.

Current nexus plugin which samza repository is using is not compatible with 
the gradle versions greater than 3.0.

For doing the ligradle migration at linkedin, we need to update the nexus 
gradle plugin to the latest version. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shanthoosh/samza master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/426.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #426


commit 24f9094a5f9efaa19ff6f928799af7b6eb3e46a3
Author: Shanthoosh Venkataraman 
Date:   2018-02-22T19:54:43Z

SAMZA-1593: Upgrade gradle nexus plugin.




> Upgrade gradle nexus plugin.
> 
>
> Key: SAMZA-1593
> URL: https://issues.apache.org/jira/browse/SAMZA-1593
> Project: Samza
>  Issue Type: Improvement
>Reporter: Shanthoosh Venkataraman
>Assignee: Shanthoosh Venkataraman
>Priority: Major
>
> Current nexus plugin which samza repository is using is not compatible with 
> the gradle versions greater than 3.0.
> For doing the ligradle jvm migration at linkedin, we need to update the nexus 
> gradle plugin to the latest version. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1460) StreamAppender does not explicitly create logging topic

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368047#comment-16368047
 ] 

ASF GitHub Bot commented on SAMZA-1460:
---

GitHub user dnishimura opened a pull request:

https://github.com/apache/samza/pull/423

SAMZA-1460: StreamAppender does not explicitly create logging topic

Creates the StreamAppender stream explicitly instead of relying on auto 
stream creation.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dnishimura/samza 
samza-1460-streamappender-create-logging-topic

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/423.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #423


commit 9c6437fc1b450c8b60ff6543a180d3b4ae929d4e
Author: Daniel Nishimura 
Date:   2018-02-17T00:27:12Z

In StreamAppender, explicitly create the stream in the setup.




> StreamAppender does not explicitly create logging topic
> ---
>
> Key: SAMZA-1460
> URL: https://issues.apache.org/jira/browse/SAMZA-1460
> Project: Samza
>  Issue Type: Bug
>Reporter: Yi Pan (Data Infrastructure)
>Priority: Major
>
> In StreamAppender, the logging topic is not explicitly created. We will be 
> relying on Kafka cluster topic auto-creation feature to create the logging 
> topic:
> {code}
>   protected void setupSystem() {
> config = getConfig();
> SystemFactory systemFactory = null;
> Log4jSystemConfig log4jSystemConfig = new Log4jSystemConfig(config);
> if (streamName == null) {
>   streamName = getStreamName(log4jSystemConfig.getJobName(), 
> log4jSystemConfig.getJobId());
> }
> String systemName = log4jSystemConfig.getSystemName();
> String systemFactoryName = log4jSystemConfig.getSystemFactory(systemName);
> if (systemFactoryName != null) {
>   systemFactory = Util.getObj(systemFactoryName);
> } else {
>   throw new SamzaException("Could not figure out \"" + systemName + "\" 
> system factory for log4j StreamAppender to use");
> }
> setSerde(log4jSystemConfig, systemName, streamName);
> systemProducer = systemFactory.getProducer(systemName, config, new 
> MetricsRegistryMap());
> systemStream = new SystemStream(systemName, streamName);
> systemProducer.register(SOURCE);
> systemProducer.start();
> log.info(SOURCE + " has been registered in " + systemName + ". So all the 
> logs will be sent to " + streamName
> + " in " + systemName + ". Logs are partitioned by " + key);
>   }
> {code}
> It would be better to explicitly create the logging topic via 
> SystemAdmin.createStream() to work w/ logging systems that do not support 
> topic auto-creation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1555) Move creation of checkpoint and changelog streams to the Job Coordinators

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367939#comment-16367939
 ] 

ASF GitHub Bot commented on SAMZA-1555:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/413


> Move creation of checkpoint and changelog streams to the Job Coordinators
> -
>
> Key: SAMZA-1555
> URL: https://issues.apache.org/jira/browse/SAMZA-1555
> Project: Samza
>  Issue Type: Improvement
>Affects Versions: 0.15.0
>Reporter: Daniel Nishimura
>Assignee: Daniel Nishimura
>Priority: Major
>
> This is an initial step ensuring components are responsible for only what 
> they should be responsible for. For example, currently the JobModelManager 
> creates the changelog stream. The JobModelManager should only create and 
> manage the JobModel.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1588) Add random jitter to monitor’s scheduling interval.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367704#comment-16367704
 ] 

ASF GitHub Bot commented on SAMZA-1588:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/422


> Add random jitter to monitor’s scheduling interval.
> ---
>
> Key: SAMZA-1588
> URL: https://issues.apache.org/jira/browse/SAMZA-1588
> Project: Samza
>  Issue Type: Improvement
>Reporter: Shanthoosh Venkataraman
>Assignee: Shanthoosh Venkataraman
>Priority: Minor
>
> We’ve observed in LinkedIn execution environments that, all the monitors 
> running on the YARN node-manager machines hitting an external service at the 
> same time based upon the configured monitor scheduling interval.
> To eliminate unnecessary monitor execution spike and congestion caused to an 
> external service at the same time, it’s essential to add a random jitter to 
> the monitor scheduling interval.
> Random jitter will be added to monitor scheduling interval based upon a 
> boolean monitor configuration
> Creating this ticket for book-keeping purposes to track these changes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1588) Add random jitter to monitor’s scheduling interval.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366249#comment-16366249
 ] 

ASF GitHub Bot commented on SAMZA-1588:
---

GitHub user shanthoosh opened a pull request:

https://github.com/apache/samza/pull/422

SAMZA-1588: Add random jitter to monitor’s scheduling interval.

We’ve observed in LinkedIn execution environments that, all the monitors 
running on the YARN node-manager machines hitting an external service at the 
same time based upon the configured monitor scheduling interval.

To eliminate unnecessary monitor execution spike and congestion caused to 
an external service at the same time, it’s essential to add a random jitter to 
the monitor scheduling interval.

Random jitter will be added to monitor scheduling interval based upon a 
boolean configuration.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shanthoosh/samza SAMZA-1588

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/422.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #422


commit 9ff3140156546157fef3a12255beeb05c1feccac
Author: Shanthoosh Venkataraman 
Date:   2018-02-15T20:40:12Z

SAMZA-1588: Add random jitter to monitor’s scheduling interval.




> Add random jitter to monitor’s scheduling interval.
> ---
>
> Key: SAMZA-1588
> URL: https://issues.apache.org/jira/browse/SAMZA-1588
> Project: Samza
>  Issue Type: Improvement
>Reporter: Shanthoosh Venkataraman
>Assignee: Shanthoosh Venkataraman
>Priority: Minor
>
> We’ve observed in LinkedIn execution environments that, all the monitors 
> running on the YARN node-manager machines hitting an external service at the 
> same time based upon the configured monitor scheduling interval.
> To eliminate unnecessary monitor execution spike and congestion caused to an 
> external service at the same time, it’s essential to add a random jitter to 
> the monitor scheduling interval.
> Random jitter will be added to monitor scheduling interval based upon a 
> boolean monitor configuration
> Creating this ticket for book-keeping purposes to track these changes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1568) Handle ZkInterruptedException in zkclient.close.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363437#comment-16363437
 ] 

ASF GitHub Bot commented on SAMZA-1568:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/416


> Handle ZkInterruptedException in zkclient.close.
> 
>
> Key: SAMZA-1568
> URL: https://issues.apache.org/jira/browse/SAMZA-1568
> Project: Samza
>  Issue Type: Bug
>Reporter: Shanthoosh Venkataraman
>Assignee: Shanthoosh Venkataraman
>Priority: Major
>
> When zookeeper session failures occur in a stream processor,   leaves the 
> group(zkClient is closed) and joins the group again.
> The last step in that shutdown sequence is zkClient.close(). In some 
> scenarios, it throws the following exception, 
> {code:java}
>     org.I0Itec.zkclient.exception.ZkInterruptedException: 
> java.lang.InterruptedException
>     at org.I0Itec.zkclient.ZkClient.close(ZkClient.java:1278)
>     at org.apache.samza.zk.ZkControllerImpl.stop(ZkControllerImpl.java:92)
>     at org.apache.samza.zk.ZkJobCoordinator.stop(ZkJobCoordinator.java:141)
> {code}
> In existing implementation this is not handled, there by killing the stream 
> processor.  The following codepath triggers this exception:
> {code:java}
> StreamProcessor.stop -> ZkJobCoordinator.stop() ->  zkController.stop() -> 
> zkUtils.close
> {code}
> This exception causes the integration test to fail occasionally  and can 
> cause LocalApplicationRunner.waitForFinish method call to block 
> indefinitely(since this callback event success, updates the latch state 
> required for waitForFinish to end).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1572) Add fixed retries on failure in KafkaCheckpointManager

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362910#comment-16362910
 ] 

ASF GitHub Bot commented on SAMZA-1572:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/420


> Add fixed retries on failure in KafkaCheckpointManager
> --
>
> Key: SAMZA-1572
> URL: https://issues.apache.org/jira/browse/SAMZA-1572
> Project: Samza
>  Issue Type: Bug
>Reporter: Shanthoosh Venkataraman
>Assignee: Shanthoosh Venkataraman
>Priority: Major
>
> KafkaCheckpointManager.writeCheckpoint currently goes into a infinite loop 
> when an irrecoverable failure happens, this indefinitely blocks the commit 
> phase (there by preventing processing). This exception is revealed only 
> during the shutdown of the job making shutdown block indefinitely since the 
> markers for shutdown are ignored by runloop which is blocked on commit phase.
> {code:java}
> 2018/01/22 19:18:10.503 WARN [KafkaCheckpointManager]  [] Failed to write 
> checkpoint log partition entry 
> org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey@8148c5bb: 
> org.apache.samza.system.SystemProducerException: Flush failed. One or more 
> batches of messages were not sent. Retrying. 2018/01/22 19:18:10.604 WARN 
> [KafkaCheckpointManager]  [] Failed to write checkpoint log partition entry 
> org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey@8148c5bb: 
> org.apache.samza.system.SystemProducerException: Producer was unable to 
> recover from previous exceptio 2018/01/22 19:18:10.804 WARN 
> [KafkaCheckpointManager]  [] Failed to write checkpoint log partition entry 
> org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey@8148c5bb: 
> org.apache.samza.system.SystemProducerException: Producer was unable to 
> recover from previous exceptio 2018/01/22 19:18:11.204 WARN 
> [KafkaCheckpointManager]  [] Failed to write checkpoint log partition entry 
> org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey@8148c5bb: 
> org.apache.samza.system.SystemProducerException: Producer was unable to 
> recover from previous exceptio 2018/01/22 19:18:12.005 WARN 
> [KafkaCheckpointManager]  [] Failed to write checkpoint log partition entry 
> org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey@8148c5bb: 
> org.apache.samza.system.SystemProducerException: Producer was unable to 
> recover from previous exceptio 2018/01/22 19:18:13.605 WARN 
> [KafkaCheckpointManager]  [] Failed to write checkpoint log partition entry 
> org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey@8148c5bb: 
> org.apache.samza.system.SystemProducerException: Producer was unable to 
> recover from previous exceptio 2018/01/22 19:18:16.805 WARN 
> [KafkaCheckpointManager]  [] Failed to write checkpoint log partition entry 
> org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey@8148c5bb: 
> org.apache.samza.system.SystemProducerException: Producer was unable to 
> recover from previous exceptio 2018/01/22 19:18:23.205 WARN 
> [KafkaCheckpointManager]  [] Failed to write checkpoint log partition entry 
> org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey@8148c5bb: 
> org.apache.samza.system.SystemProducerException: Producer was unable to 
> recover from previous exceptio 2018/01/22 19:18:33.206 WARN 
> [KafkaCheckpointManager]  [] Failed to write checkpoint log partition entry 
> org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey@8148c5bb: 
> org.apache.samza.system.SystemProducerException: Producer was unable to 
> recover from previous exceptio 2018/01/22 19:18:43.206 WARN 
> [KafkaCheckpointManager]  [] Failed to write checkpoint log partition entry 
> org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey@8148c5bb: 
> org.apache.samza.system.SystemProducerException: Producer was unable to 
> recover from previous exceptio 2018/01/22 19:18:53.206 WARN 
> [KafkaCheckpointManager]  [] Failed to write checkpoint log partition entry 
> org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey@8148c5bb: 
> org.apache.samza.system.SystemProducerException: Producer was unable to 
> recover from previous exceptio 2018/01/22 19:19:03.207 WARN 
> [KafkaCheckpointManager]  [] Failed to write checkpoint log partition entry 
> org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey@8148c5bb: 
> org.apache.samza.system.SystemProducerException: Producer was unable to 
> recover from previous exceptio 2018/01/22 19:19:13.207 WARN 
> [KafkaCheckpointManager]  [] Failed to write checkpoint log partition entry 
> org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey@8148c5bb: 
> org.apache.samza.system.SystemProducerException: Producer was unable to 
> recover from previous exceptio 2018/01/22 19:19:23.207 WARN 
> [KafkaCheckpointManager]  [] Failed to write checkpoint log partition entry 
> 

[jira] [Commented] (SAMZA-1572) Add fixed retries on failure in KafkaCheckpointManager

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356032#comment-16356032
 ] 

ASF GitHub Bot commented on SAMZA-1572:
---

GitHub user shanthoosh opened a pull request:

https://github.com/apache/samza/pull/420

SAMZA-1572: Add fixed retries on failure in KafkaCheckpointManager.

KafkaCheckpointManager.writeCheckpoint currently goes into a infinite loop 
when an irrecoverable failure happens, this indefinitely blocks the commit 
phase (there by preventing processing). Added finite retries (50), which would 
retry for fixed time in case of failure before giving up. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shanthoosh/samza 
add_fixed_retries_in_kafka_checkpoint_manager

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/420.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #420


commit 8b98814e9d96b17a5772d079c20832f6f094640e
Author: Shanthoosh Venkataraman 
Date:   2018-01-25T22:10:28Z

SAMZA-1572: Add fixed retries on failure in KafkaCheckpointManager.




> Add fixed retries on failure in KafkaCheckpointManager
> --
>
> Key: SAMZA-1572
> URL: https://issues.apache.org/jira/browse/SAMZA-1572
> Project: Samza
>  Issue Type: Bug
>Reporter: Shanthoosh Venkataraman
>Assignee: Shanthoosh Venkataraman
>Priority: Major
>
> KafkaCheckpointManager.writeCheckpoint currently goes into a infinite loop 
> when an irrecoverable failure happens, this indefinitely blocks the commit 
> phase (there by preventing processing). This exception is revealed only 
> during the shutdown of the job making shutdown block indefinitely since the 
> markers for shutdown are ignored by runloop which is blocked on commit phase.
> {code:java}
> 2018/01/22 19:18:10.503 WARN [KafkaCheckpointManager]  [] Failed to write 
> checkpoint log partition entry 
> org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey@8148c5bb: 
> org.apache.samza.system.SystemProducerException: Flush failed. One or more 
> batches of messages were not sent. Retrying. 2018/01/22 19:18:10.604 WARN 
> [KafkaCheckpointManager]  [] Failed to write checkpoint log partition entry 
> org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey@8148c5bb: 
> org.apache.samza.system.SystemProducerException: Producer was unable to 
> recover from previous exceptio 2018/01/22 19:18:10.804 WARN 
> [KafkaCheckpointManager]  [] Failed to write checkpoint log partition entry 
> org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey@8148c5bb: 
> org.apache.samza.system.SystemProducerException: Producer was unable to 
> recover from previous exceptio 2018/01/22 19:18:11.204 WARN 
> [KafkaCheckpointManager]  [] Failed to write checkpoint log partition entry 
> org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey@8148c5bb: 
> org.apache.samza.system.SystemProducerException: Producer was unable to 
> recover from previous exceptio 2018/01/22 19:18:12.005 WARN 
> [KafkaCheckpointManager]  [] Failed to write checkpoint log partition entry 
> org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey@8148c5bb: 
> org.apache.samza.system.SystemProducerException: Producer was unable to 
> recover from previous exceptio 2018/01/22 19:18:13.605 WARN 
> [KafkaCheckpointManager]  [] Failed to write checkpoint log partition entry 
> org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey@8148c5bb: 
> org.apache.samza.system.SystemProducerException: Producer was unable to 
> recover from previous exceptio 2018/01/22 19:18:16.805 WARN 
> [KafkaCheckpointManager]  [] Failed to write checkpoint log partition entry 
> org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey@8148c5bb: 
> org.apache.samza.system.SystemProducerException: Producer was unable to 
> recover from previous exceptio 2018/01/22 19:18:23.205 WARN 
> [KafkaCheckpointManager]  [] Failed to write checkpoint log partition entry 
> org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey@8148c5bb: 
> org.apache.samza.system.SystemProducerException: Producer was unable to 
> recover from previous exceptio 2018/01/22 19:18:33.206 WARN 
> [KafkaCheckpointManager]  [] Failed to write checkpoint log partition entry 
> org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey@8148c5bb: 
> org.apache.samza.system.SystemProducerException: Producer was unable to 
> recover from previous exceptio 2018/01/22 19:18:43.206 WARN 
> [KafkaCheckpointManager]  [] Failed to write checkpoint log partition entry 
> org.apache.samza.checkpoint.kafka.KafkaCheckpointLogKey@8148c5bb: 
> org.apache.samza.system.SystemProducerException: Producer was 

[jira] [Commented] (SAMZA-1498) Support arbitrary system clock timer in operators

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354890#comment-16354890
 ] 

ASF GitHub Bot commented on SAMZA-1498:
---

GitHub user xinyuiscool opened a pull request:

https://github.com/apache/samza/pull/419

SAMZA-1498: Support arbitrary system clock timer in operators

This patch adds the capability to register arbitrary timers for both 
high-level and low-level api.  
For high-level, InitableFunction will pass the TimerRegistry to user 
through the new OpContext, and user will implement the TimerFunction to get 
timer notifications.
For low-level api, user can register timer in the TaskContext, and then 
implements the TimerCallback for specific timer actions.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xinyuiscool/samza SAMZA-1498

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/419.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #419


commit 35e93685e6c8955d19ad84d7255bee4a85b3d041
Author: xiliu 
Date:   2018-02-01T21:57:48Z

SAMZA-1498: Support arbitrary system clock timer in operators

commit a239b84dd75c018db2b14cbf138501e017957434
Author: xiliu 
Date:   2018-02-02T21:28:11Z

Fix checkstyle issues

commit 3be14b394e029016014819e15d12d52f86be6fbc
Author: xiliu 
Date:   2018-02-02T23:28:32Z

Change init function interface

commit 2f4bf429728e4dfdc1d8c51968ea774d117d5e87
Author: xiliu 
Date:   2018-02-02T23:30:26Z

Some factoring on the init()

commit f9492cff1b636845e5ce15f22235e6568b952cd0
Author: xiliu 
Date:   2018-02-03T00:44:00Z

Another adjustment of the API

commit e13d077c7ee0d5a0be3f79469563a2bdb343bfe5
Author: xiliu 
Date:   2018-02-07T01:50:55Z

Java docs and unit tests

commit e6c1918cf076751fbdc53ad68a36656a212a4afb
Author: xiliu 
Date:   2018-02-07T02:37:02Z

Merge branch 'master' into SAMZA-1498

Conflicts:

samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala




> Support arbitrary system clock timer in operators
> -
>
> Key: SAMZA-1498
> URL: https://issues.apache.org/jira/browse/SAMZA-1498
> Project: Samza
>  Issue Type: Bug
>Reporter: Prateek Maheshwari
>Assignee: Xinyu Liu
>Priority: Minor
> Fix For: 0.15.0
>
>
> We need the capability of supporting any system-clock timers for user code. 
> So user can register any timer as they like and they will get a callback in 
> their functions. Internally we will be able to register the timer inside a 
> scheduler and AsyncRunLoop will query the timers and fire the callback 
> accordingly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1105) Upgrade Samza's Kafka client version to 0.10.2.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352990#comment-16352990
 ] 

ASF GitHub Bot commented on SAMZA-1105:
---

Github user garlicbulb-puzhuo closed the pull request at:

https://github.com/apache/samza/pull/63


> Upgrade Samza's Kafka client version to 0.10.2.0
> 
>
> Key: SAMZA-1105
> URL: https://issues.apache.org/jira/browse/SAMZA-1105
> Project: Samza
>  Issue Type: Task
>Reporter: Chen Song
>Assignee: Chen Song
>Priority: Minor
> Fix For: 0.13.0
>
>
> Upgrade Samza's kafka client version to 0.10.2.0.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1579) RemoteApplicationRunner should allow user to kill app or query app status

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350964#comment-16350964
 ] 

ASF GitHub Bot commented on SAMZA-1579:
---

Github user lindong28 closed the pull request at:

https://github.com/apache/samza/pull/418


> RemoteApplicationRunner should allow user to kill app or query app status
> -
>
> Key: SAMZA-1579
> URL: https://issues.apache.org/jira/browse/SAMZA-1579
> Project: Samza
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1489) TaskInstance should commit offset before it closes() if auto commit is enabled

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349182#comment-16349182
 ] 

ASF GitHub Bot commented on SAMZA-1489:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/417


> TaskInstance should commit offset before it closes() if auto commit is enabled
> --
>
> Key: SAMZA-1489
> URL: https://issues.apache.org/jira/browse/SAMZA-1489
> Project: Samza
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Prateek Maheshwari
>Priority: Major
> Fix For: 0.15.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1552) Host affinity improvements - Improve matching of hosts to allocated resources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348040#comment-16348040
 ] 

ASF GitHub Bot commented on SAMZA-1552:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/401


> Host affinity improvements - Improve matching of hosts to allocated resources
> -
>
> Key: SAMZA-1552
> URL: https://issues.apache.org/jira/browse/SAMZA-1552
> Project: Samza
>  Issue Type: Bug
>Reporter: Abhishek Shivanna
>Assignee: Jagadish
>Priority: Major
>
> Kudos to [~abkshvn] for observing this!
> We have observed host-affinity not being honored for some containers in very 
> large jobs. When Yarn allocates more resources than what Samza requested on a 
> specific host, the extra resources are added to a spare-pool called the 
> "ANY_HOST Buffer". Later, when there is a resource request for the same host 
> from Samza and Yarn does not return resources, we don't leverage the 
> spare-pool of previously returned resources in that host. 
> This problem is specially pronounced in clusters that are heavily loaded in 
> cpu, and memory where allocations need to satisfy both cpu and memory 
> requirements of available hosts (Often, hosts have cpu but not memory or 
> vice-versa). If there are a lot of container failures on a particular host in 
> the midst of allocation, it further aggravates this problem.
> The fix is as follows:
> Check if there are available containers in the buffer corresponding to our 
> preferred host. If not, we should also scan the ANY-HOST buffer for matched 
> containers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1579) RemoteApplicationRunner should allow user to kill app or query app status

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345933#comment-16345933
 ] 

ASF GitHub Bot commented on SAMZA-1579:
---

GitHub user lindong28 opened a pull request:

https://github.com/apache/samza/pull/418

SAMZA-1579; RemoteApplicationRunner should allow user to kill app or query 
app status



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lindong28/samza SAMZA-1579

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/418.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #418


commit 5687213a7290030e5bf630c14f199173c13d2339
Author: Dong Lin 
Date:   2018-01-30T22:31:54Z

SAMZA-1579; RemoteApplicationRunner should allow user to kill app or query 
app status




> RemoteApplicationRunner should allow user to kill app or query app status
> -
>
> Key: SAMZA-1579
> URL: https://issues.apache.org/jira/browse/SAMZA-1579
> Project: Samza
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1489) TaskInstance should commit offset before it closes() if auto commit is enabled

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345884#comment-16345884
 ] 

ASF GitHub Bot commented on SAMZA-1489:
---

GitHub user lindong28 opened a pull request:

https://github.com/apache/samza/pull/417

SAMZA-1489; TaskInstance should commit offset before it closes() if auto 
commit is enabled



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lindong28/samza SAMZA-1489

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/417.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #417


commit 35190f9b4a15646a7bac3d9afa6dfdd2c02de01b
Author: Dong Lin 
Date:   2018-01-30T21:59:23Z

SAMZA-1489; TaskInstance should commit offset before it closes() if auto 
commit is enabled




> TaskInstance should commit offset before it closes() if auto commit is enabled
> --
>
> Key: SAMZA-1489
> URL: https://issues.apache.org/jira/browse/SAMZA-1489
> Project: Samza
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Prateek Maheshwari
>Priority: Major
> Fix For: 0.15.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1568) Handle ZkInterruptedException in zkclient.close.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341879#comment-16341879
 ] 

ASF GitHub Bot commented on SAMZA-1568:
---

GitHub user shanthoosh opened a pull request:

https://github.com/apache/samza/pull/416

SAMZA-1568: Handle ZkInterruptedException in zkclient.close.

When zookeeper session failures occur in a stream processor,   leaves the 
group(zkClient is closed) and joins the group again.

The last step in that shutdown sequence is zkClient.close(). In some 
scenarios, it throws the following exception, 

org.I0Itec.zkclient.exception.ZkInterruptedException: 
java.lang.InterruptedException
at org.I0Itec.zkclient.ZkClient.close(ZkClient.java:1278)
at org.apache.samza.zk.ZkControllerImpl.stop(ZkControllerImpl.java:92)

at org.apache.samza.zk.ZkJobCoordinator.stop(ZkJobCoordinator.java:141)
In existing implementation this is not handled, there by killing the stream 
processor.  The following codepath triggers this exception:

`StreamProcessor.stop -> ZkJobCoordinator.stop() ->  zkController.stop() -> 
zkUtils.close`

This exception causes the integration test to fail occasionally  and can 
cause LocalApplicationRunner.waitForFinish method call to block 
indefinitely(since this callback event success, updates the latch state 
required for waitForFinish to end).



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shanthoosh/samza zk_utils_close

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/416.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #416


commit 2542a0ad1bee19927761eaf0171a6f637f21ac3a
Author: Shanthoosh Venkataraman 
Date:   2018-01-25T19:44:55Z

SAMZA-1568: Handle ZkInterruptedException in zkclient.close.




> Handle ZkInterruptedException in zkclient.close.
> 
>
> Key: SAMZA-1568
> URL: https://issues.apache.org/jira/browse/SAMZA-1568
> Project: Samza
>  Issue Type: Bug
>Reporter: Shanthoosh Venkataraman
>Assignee: Shanthoosh Venkataraman
>Priority: Major
>
> When zookeeper session failures occur in a stream processor,   leaves the 
> group(zkClient is closed) and joins the group again.
> The last step in that shutdown sequence is zkClient.close(). In some 
> scenarios, it throws the following exception, 
> {code:java}
>     org.I0Itec.zkclient.exception.ZkInterruptedException: 
> java.lang.InterruptedException
>     at org.I0Itec.zkclient.ZkClient.close(ZkClient.java:1278)
>     at org.apache.samza.zk.ZkControllerImpl.stop(ZkControllerImpl.java:92)
>     at org.apache.samza.zk.ZkJobCoordinator.stop(ZkJobCoordinator.java:141)
> {code}
> In existing implementation this is not handled, there by killing the stream 
> processor.  The following codepath triggers this exception:
> {code:java}
> StreamProcessor.stop -> ZkJobCoordinator.stop() ->  zkController.stop() -> 
> zkUtils.close
> {code}
> This exception causes the integration test to fail occasionally  and can 
> cause LocalApplicationRunner.waitForFinish method call to block 
> indefinitely(since this callback event success, updates the latch state 
> required for waitForFinish to end).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1578) Fix watermark bug found by BEAM tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341659#comment-16341659
 ] 

ASF GitHub Bot commented on SAMZA-1578:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/415


> Fix watermark bug found by BEAM tests
> -
>
> Key: SAMZA-1578
> URL: https://issues.apache.org/jira/browse/SAMZA-1578
> Project: Samza
>  Issue Type: Bug
>Affects Versions: 0.15.0
>Reporter: Xinyu Liu
>Assignee: Xinyu Liu
>Priority: Major
> Fix For: 0.15.0
>
>
> Found a bug in the watermark propagation logic in Samza. The problem is 
> getOutputWatermark() does not return the real outputWatermark. This caused 
> problem in user override watermark function.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1578) Fix watermark bug found by BEAM tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341604#comment-16341604
 ] 

ASF GitHub Bot commented on SAMZA-1578:
---

GitHub user xinyuiscool opened a pull request:

https://github.com/apache/samza/pull/415

SAMZA-1578: Fix watermark bug found by BEAM tests

The problem is getOutputWatermark() does not return the real 
outputWatermark. This caused problem in user override watermark function.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xinyuiscool/samza SAMZA-1578

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/415.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #415


commit 392e483b08267257ff7d75a2441e8b5bc9f214a6
Author: xiliu 
Date:   2018-01-26T21:09:10Z

SAMZA-1578: Fix watermark bug found by BEAM tests




> Fix watermark bug found by BEAM tests
> -
>
> Key: SAMZA-1578
> URL: https://issues.apache.org/jira/browse/SAMZA-1578
> Project: Samza
>  Issue Type: Bug
>Affects Versions: 0.15.0
>Reporter: Xinyu Liu
>Assignee: Xinyu Liu
>Priority: Major
> Fix For: 0.15.0
>
>
> Found a bug in the watermark propagation logic in Samza. The problem is 
> getOutputWatermark() does not return the real outputWatermark. This caused 
> problem in user override watermark function.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1548) Add start() and stop() to SystemAdmin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340404#comment-16340404
 ] 

ASF GitHub Bot commented on SAMZA-1548:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/397


> Add start() and stop() to SystemAdmin
> -
>
> Key: SAMZA-1548
> URL: https://issues.apache.org/jira/browse/SAMZA-1548
> Project: Samza
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
>Priority: Major
> Fix For: 0.15.0
>
>
> This patch adds start() and stop() to SystemAdmin interface. This can be 
> useful for e.g. kafka.admin.AdminClient which needs to be started before it 
> can be used.
> Since we add this method in interface and expect AdminClient to be stateful 
> and probably has its own thread, there will be higher cost to instantiate a 
> new SystemAdmin. Thus we probably want to re-use the SystemAdmin instances 
> instead of creating SystemAdmin on demand when needed. Therefore, this patch 
> also adds SystemAdmins class to help manage a map from system to SystemAdmin, 
> similar to the existing SystemProducers class in Samza.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1557) Broadcast operator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338345#comment-16338345
 ] 

ASF GitHub Bot commented on SAMZA-1557:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/410


> Broadcast operator
> --
>
> Key: SAMZA-1557
> URL: https://issues.apache.org/jira/browse/SAMZA-1557
> Project: Samza
>  Issue Type: Bug
>Affects Versions: 0.15.0
>Reporter: Xinyu Liu
>Assignee: Xinyu Liu
>Priority: Major
> Fix For: 0.15.0
>
>
> Broadcast operator to broadcast messages to all tasks. It's the counterpart 
> of the Samza broadcast stream in low level api, and will be used by BEAM 
> runner to broadcast views as side input to other part of the pipeline.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1555) Move creation of checkpoint and changelog streams to the Job Coordinators

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335988#comment-16335988
 ] 

ASF GitHub Bot commented on SAMZA-1555:
---

GitHub user dnishimura opened a pull request:

https://github.com/apache/samza/pull/413

SAMZA-1555: Move creation of checkpoint and changelog streams to the Job 
Coordinators

**Overview**
The purpose of this PR is to consolidate the creation of the changelog and 
checkpoint streams into the JobCoordinators. In the current state, the 
changelog stream is created from the JobModelManager and the checkpoint stream 
is created within the OffsetManager. The issue with creating the checkpoint in 
the OffsetManager is that the first call happens from the first SamzaContainer 
that runs and each subsequent SamzaContainer run will attempt to create the 
checkpoint stream.

**Motivations**
There are three driving forces for this refactoring. The first motivation 
is to assign the creation of the changelog and checkpoint streams to the 
JobCoordinators where it is most appropriate. This was discussed in more detail 
with @nickpan47  . The second motivation is to have any potential failure to 
stream creation happen no later than during job coordination. The third 
motivation is to accommodate future security work to provide a robust way to 
set ACLs on streams.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dnishimura/samza 
samza-1555-move-changelog-checkpoint-creation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/413.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #413


commit 971fa91afb7f57fc8c099a7fb17e7bc96bc826c2
Author: Daniel Nishimura 
Date:   2018-01-16T21:20:07Z

Move the responsibility of changelog and checkpoint stream creation to the 
job coordinators.




> Move creation of checkpoint and changelog streams to the Job Coordinators
> -
>
> Key: SAMZA-1555
> URL: https://issues.apache.org/jira/browse/SAMZA-1555
> Project: Samza
>  Issue Type: Improvement
>Affects Versions: 0.15.0
>Reporter: Daniel Nishimura
>Assignee: Daniel Nishimura
>Priority: Major
>
> This is an initial step ensuring components are responsible for only what 
> they should be responsible for. For example, currently the JobModelManager 
> creates the changelog stream. The JobModelManager should only create and 
> manage the JobModel.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1562) TaskStorageManager should delete any local store it cannot open.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335345#comment-16335345
 ] 

ASF GitHub Bot commented on SAMZA-1562:
---

GitHub user jmakes opened a pull request:

https://github.com/apache/samza/pull/412

SAMZA-1562: TaskStorageManager should delete any local store it canno…

…t open.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jmakes/samza samza-1562

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/412.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #412


commit f2a687c46846392d20eccba21ac385ba4758b836
Author: Jacob Maes 
Date:   2018-01-23T03:21:52Z

SAMZA-1562: TaskStorageManager should delete any local store it cannot open.




> TaskStorageManager should delete any local store it cannot open.
> 
>
> Key: SAMZA-1562
> URL: https://issues.apache.org/jira/browse/SAMZA-1562
> Project: Samza
>  Issue Type: Bug
>Reporter: Jake Maes
>Assignee: Jake Maes
>Priority: Major
>
> If an exception occurs while validating/opening the local store, the 
> TaskStorageManager should delete the whole logged store directory. Otherwise 
> the task will never successfully run on that host again. 
>  
> Examples
> {noformat}
> .../Partition_349/000512.sst: No such file or directory
> at 
> org.apache.samza.storage.kv.RocksDbKeyValueStore$.openDB(RocksDbKeyValueStore.scala:87)
>  at 
> org.apache.samza.storage.kv.RocksDbKeyValueStore.db$lzycompute(RocksDbKeyValueStore.scala:109)
>  at 
> org.apache.samza.storage.kv.RocksDbKeyValueStore.db(RocksDbKeyValueStore.scala:109)
>  at 
> org.apache.samza.storage.kv.RocksDbKeyValueStore.putAll(RocksDbKeyValueStore.scala:166)
>  at 
> org.apache.samza.storage.kv.KeyValueStorageEngine.restore(KeyValueStorageEngine.scala:129)
>  at 
> org.apache.samza.storage.TaskStorageManager$$anonfun$restoreStores$3.apply(TaskStorageManager.scala:256)
>  at 
> org.apache.samza.storage.TaskStorageManager$$anonfun$restoreStores$3.apply(TaskStorageManager.scala:250)
>  at 
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
>  at scala.collection.immutable.Map$Map3.foreach(Map.scala:154)
>  at 
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
>  at 
> org.apache.samza.storage.TaskStorageManager.restoreStores(TaskStorageManager.scala:250)
>  at 
> org.apache.samza.storage.TaskStorageManager.init(TaskStorageManager.scala:81)
>  at 
> org.apache.samza.container.TaskInstance.startStores(TaskInstance.scala:116)
>  at 
> org.apache.samza.container.SamzaContainer$$anonfun$startStores$2.apply(SamzaContainer.scala:866)
>  at 
> org.apache.samza.container.SamzaContainer$$anonfun$startStores$2.apply(SamzaContainer.scala:864)
>  at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>  at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>  at scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:206)
>  at 
> org.apache.samza.container.SamzaContainer.startStores(SamzaContainer.scala:864)
>  at org.apache.samza.container.SamzaContainer.run(SamzaContainer.scala:685)
>  at 
> org.apache.samza.runtime.LocalContainerRunner.run(LocalContainerRunner.java:104)
> {noformat}
> and
> {noformat}
> 2018-01-22 16:39:29.824 [main] SamzaContainer [ERROR] Caught exception/error 
> while initializing container.
> java.lang.NullPointerException
> at org.apache.samza.util.Util$.readDataFromFile(Util.scala:318)
> at 
> org.apache.samza.storage.TaskStorageManager.org$apache$samza$storage$TaskStorageManager$$readOffsetFile(TaskStorageManager.scala:200)
> at 
> org.apache.samza.storage.TaskStorageManager.isOffsetFileValid(TaskStorageManager.scala:163)
> at 
> org.apache.samza.storage.TaskStorageManager.org$apache$samza$storage$TaskStorageManager$$isLoggedStoreValid(TaskStorageManager.scala:124)
> at 
> org.apache.samza.storage.TaskStorageManager$$anonfun$cleanBaseDirs$2.apply(TaskStorageManager.scala:101)
> at 
> org.apache.samza.storage.TaskStorageManager$$anonfun$cleanBaseDirs$2.apply(TaskStorageManager.scala:88)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at scala.collection.MapLike$DefaultKeySet.foreach(MapLike.scala:174)
> at 
> org.apache.samza.storage.TaskStorageManager.cleanBaseDirs(TaskStorageManager.scala:88)
> at 
> org.apache.samza.storage.TaskStorageManager.init(TaskStorageManager.scala:77)
> at org.apache.samza.container.TaskInstance.startStores(TaskInstance.scala:116)
> at 
> 

[jira] [Commented] (SAMZA-1561) JobModel upgrade consistency problem.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334778#comment-16334778
 ] 

ASF GitHub Bot commented on SAMZA-1561:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/409


> JobModel upgrade consistency problem.
> -
>
> Key: SAMZA-1561
> URL: https://issues.apache.org/jira/browse/SAMZA-1561
> Project: Samza
>  Issue Type: Bug
>Reporter: Shanthoosh Venkataraman
>Assignee: Shanthoosh Venkataraman
>Priority: Major
>
> JobModel upgrade sequence is the following: 
> A. Read previousJobModelVersion from JobModelBasePath/jobModelVersion.
> B. Publish the new JobModel with version (previousJobModelVersion + 1) to 
> JobModelBasePath/jobmodels.
> C. Create a barrier with version (previousJobModelVersion + 1).
> D. Update jobModelVersion path with value (previousJobModelVersion + 1).
> Followers watch on jobModelVersion path for JobModel upgrades.
> If the leader dies before executing the last step of the upgrade sequence, 
> then any processor elected as leader will be unable to publish the new 
> JobModel and will fail with ZkNodeExistsException (For instance, 
> previousJobModel version is 2 of a processor group [P1, P2]. P1 is the leader 
> and it created zkNode jobModelBasePath/jobModels/3 for publishing jobModel 
> and dies without upgrading jobModelVersion path(which stays as 2). If P2 
> becomes leader, it will generate the jobModel version and try to create node 
> jobModelBasePath/jobModels/3 and will fail).
> This behavior was observed during the testing in one of samza standalone 
> jobs. 
> JobModelBasePath/jobModels is the source of truth for the latest 
> jobModelVersion in a processor group. By maintaining it in a separate 
> zookeeper node and not having the capability to do atomic upgrades we run 
> into this consistency problem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1557) Broadcast operator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333042#comment-16333042
 ] 

ASF GitHub Bot commented on SAMZA-1557:
---

GitHub user xinyuiscool opened a pull request:

https://github.com/apache/samza/pull/410

SAMZA-1557: Broadcast operator

This patch adds Broadcast operator that allows broadcasting messages to all 
tasks. It's the counterpart of the Samza broadcast stream in low level api, and 
will be used by BEAM runner to broadcast views as side input to other part of 
the pipeline.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xinyuiscool/samza SAMZA-1557

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/410.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #410


commit 58d7c37d17dd144c4d009311811e4e77ce19d1f0
Author: xiliu 
Date:   2018-01-17T02:12:37Z

SAMZA-1557: Broadcast operator

commit f3f40f6b5af957f9c9fc6e35e4fc1eb50672d0c0
Author: xiliu 
Date:   2018-01-18T23:09:14Z

Add integration tests

commit a1c7100d243f22dae1de3d4e8004043dc79bb681
Author: xiliu 
Date:   2018-01-18T23:47:21Z

Small enhancements

commit 23bcada447d19fb2103db61ac2b0daa632ff6707
Author: xiliu 
Date:   2018-01-20T00:04:52Z

Some minor fix to compute the output to input mapping for broadcast op




> Broadcast operator
> --
>
> Key: SAMZA-1557
> URL: https://issues.apache.org/jira/browse/SAMZA-1557
> Project: Samza
>  Issue Type: Bug
>Affects Versions: 0.15.0
>Reporter: Xinyu Liu
>Assignee: Xinyu Liu
>Priority: Major
> Fix For: 0.15.0
>
>
> Broadcast operator to broadcast messages to all tasks. It's the counterpart 
> of the Samza broadcast stream in low level api, and will be used by BEAM 
> runner to broadcast views as side input to other part of the pipeline.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1561) JobModel upgrade consistency problem.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331325#comment-16331325
 ] 

ASF GitHub Bot commented on SAMZA-1561:
---

GitHub user shanthoosh opened a pull request:

https://github.com/apache/samza/pull/409

SAMZA-1561: Fix inconsistency problem in JobModel publish.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shanthoosh/samza master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/409.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #409


commit b6ea0140fd60b7d3e2f59d5b1200c01d6c4a0b3c
Author: Shanthoosh Venkataraman 
Date:   2018-01-18T19:33:30Z

SAMZA-1561: Fix inconsistency problem present in publishing JobModel.




> JobModel upgrade consistency problem.
> -
>
> Key: SAMZA-1561
> URL: https://issues.apache.org/jira/browse/SAMZA-1561
> Project: Samza
>  Issue Type: Bug
>Reporter: Shanthoosh Venkataraman
>Assignee: Shanthoosh Venkataraman
>Priority: Major
>
> JobModel upgrade sequence is the following: 
> A. Read previousJobModelVersion from JobModelBasePath/jobModelVersion.
> B. Publish the new JobModel with version (previousJobModelVersion + 1) to 
> JobModelBasePath/jobmodels.
> C. Create a barrier with version (previousJobModelVersion + 1).
> D. Update jobModelVersion path with value (previousJobModelVersion + 1).
> Followers watch on jobModelVersion path for JobModel upgrades.
> If the leader dies before executing the last step of the upgrade sequence, 
> then any processor elected as leader will be unable to publish the new 
> JobModel and will fail with ZkNodeExistsException (For instance, 
> previousJobModel version is 2 of a processor group [P1, P2]. P1 is the leader 
> and it created zkNode jobModelBasePath/jobModels/3 for publishing jobModel 
> and dies without upgrading jobModelVersion path(which stays as 2). If P2 
> becomes leader, it will generate the jobModel version and try to create node 
> jobModelBasePath/jobModels/3 and will fail).
> This behavior was observed during the testing in one of samza standalone 
> jobs. 
> JobModelBasePath/jobModels is the source of truth for the latest 
> jobModelVersion in a processor group. By maintaining it in a separate 
> zookeeper node and not having the capability to do atomic upgrades we run 
> into this consistency problem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1407) upgrade junit version to 4.12

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329901#comment-16329901
 ] 

ASF GitHub Bot commented on SAMZA-1407:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/406


> upgrade junit version to 4.12 
> --
>
> Key: SAMZA-1407
> URL: https://issues.apache.org/jira/browse/SAMZA-1407
> Project: Samza
>  Issue Type: Task
>Reporter: Fred Ji
>Assignee: Fred Ji
>Priority: Major
> Fix For: 0.15.0
>
>
> There are some new testing code using assertNotEquals(). 
> This function is not supported in junit 4.8.1, but the explicit junit version 
> we have in gradle file is 4.8.1. 
> Luckily, in open source build, it does not fail because powermock 1.6.6 pulls 
> in junit 4.12 transitively. 
> In order to avoid the confusion regarding what junit version is used in test, 
> we should explicitly upgrade junit version to 4.12. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1560) Skip checkpoint validations for Key Serdes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329775#comment-16329775
 ] 

ASF GitHub Bot commented on SAMZA-1560:
---

GitHub user vjagadish1989 opened a pull request:

https://github.com/apache/samza/pull/408

SAMZA-1560: Handle key-serde errors in KafkaCheckpointManager



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vjagadish1989/samza kcm-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/408.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #408


commit 25ee459014fd4daec43013f76249217d5d849154
Author: Jagadish 
Date:   2018-01-17T23:58:49Z

Skip checkpoint validations for Key Serdes

commit 2bbd9ef56e26ca207a78a1b5e4b5dc8527e4f670
Author: Jagadish 
Date:   2018-01-18T00:44:04Z

Handle Serde errors in KafkaCheckpointManager with a validateCheckpoints 
flag




> Skip checkpoint validations for Key Serdes
> --
>
> Key: SAMZA-1560
> URL: https://issues.apache.org/jira/browse/SAMZA-1560
> Project: Samza
>  Issue Type: Bug
>Reporter: Jagadish
>Assignee: Jagadish
>Priority: Major
>
> Currently, when "validateCheckpoints" is set to false, we ignore validation 
> errors for the KafkaCheckpointManager. We should extend this behavior to also 
> handle serialization errors in checkpointKeySerde



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1558) State restore metrics should be duplicated and deprecated to avoid type conflicts

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329606#comment-16329606
 ] 

ASF GitHub Bot commented on SAMZA-1558:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/407


> State restore metrics should be duplicated and deprecated to avoid type 
> conflicts
> -
>
> Key: SAMZA-1558
> URL: https://issues.apache.org/jira/browse/SAMZA-1558
> Project: Samza
>  Issue Type: Bug
>Reporter: Jake Maes
>Assignee: Jake Maes
>Priority: Major
> Fix For: 0.15.0
>
>
> SAMZA-1356 changed the types of the restore metrics in 
> KeyValueStorageEngineMetrics to gauges, which could break the metric for 
> legacy jobs. 
> Instead, we should introduce new metrics for the gauges and deprecate the old 
> ones. 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1558) State restore metrics should be duplicated and deprecated to avoid type conflicts

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329383#comment-16329383
 ] 

ASF GitHub Bot commented on SAMZA-1558:
---

GitHub user jmakes opened a pull request:

https://github.com/apache/samza/pull/407

SAMZA-1558: State restore metrics should be duplicated and deprecated…

… to avoid type conflicts

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jmakes/samza samza-1558

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/407.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #407


commit aee0fde16d3472d4bb8f32435cf662dc4b9051ba
Author: Jacob Maes 
Date:   2018-01-17T20:13:10Z

SAMZA-1558: State restore metrics should be duplicated and deprecated to 
avoid type conflicts




> State restore metrics should be duplicated and deprecated to avoid type 
> conflicts
> -
>
> Key: SAMZA-1558
> URL: https://issues.apache.org/jira/browse/SAMZA-1558
> Project: Samza
>  Issue Type: Bug
>Reporter: Jake Maes
>Assignee: Jake Maes
>Priority: Major
> Fix For: 0.15.0
>
>
> SAMZA-1356 changed the types of the restore metrics in 
> KeyValueStorageEngineMetrics to gauges, which could break the metric for 
> legacy jobs. 
> Instead, we should introduce new metrics for the gauges and deprecate the old 
> ones. 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1407) upgrade junit version to 4.12

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329292#comment-16329292
 ] 

ASF GitHub Bot commented on SAMZA-1407:
---

GitHub user fredji97 opened a pull request:

https://github.com/apache/samza/pull/406

SAMZA-1407 upgrade junit version to 4.12

"./gradlew clean check" passed

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/fredji97/samza junit4_12_new

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/406.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #406


commit f34ddd65956b1ecb66925cf6864d781f37f4a7f5
Author: Fred Ji 
Date:   2018-01-17T19:25:07Z

SAMZA-1407 upgrade junit version to 4.12




> upgrade junit version to 4.12 
> --
>
> Key: SAMZA-1407
> URL: https://issues.apache.org/jira/browse/SAMZA-1407
> Project: Samza
>  Issue Type: Task
>Reporter: Fred Ji
>Assignee: Fred Ji
>Priority: Major
> Fix For: 0.15.0
>
>
> There are some new testing code using assertNotEquals(). 
> This function is not supported in junit 4.8.1, but the explicit junit version 
> we have in gradle file is 4.8.1. 
> Luckily, in open source build, it does not fail because powermock 1.6.6 pulls 
> in junit 4.12 transitively. 
> In order to avoid the confusion regarding what junit version is used in test, 
> we should explicitly upgrade junit version to 4.12. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1407) upgrade junit version to 4.12

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329252#comment-16329252
 ] 

ASF GitHub Bot commented on SAMZA-1407:
---

Github user fredji97 closed the pull request at:

https://github.com/apache/samza/pull/373


> upgrade junit version to 4.12 
> --
>
> Key: SAMZA-1407
> URL: https://issues.apache.org/jira/browse/SAMZA-1407
> Project: Samza
>  Issue Type: Task
>Reporter: Fred Ji
>Assignee: Fred Ji
>Priority: Major
> Fix For: 0.15.0
>
>
> There are some new testing code using assertNotEquals(). 
> This function is not supported in junit 4.8.1, but the explicit junit version 
> we have in gradle file is 4.8.1. 
> Luckily, in open source build, it does not fail because powermock 1.6.6 pulls 
> in junit 4.12 transitively. 
> In order to avoid the confusion regarding what junit version is used in test, 
> we should explicitly upgrade junit version to 4.12. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1523) Cleanup table entries before shutting down the processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329131#comment-16329131
 ] 

ASF GitHub Bot commented on SAMZA-1523:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/379


> Cleanup table entries before shutting down the processor
> 
>
> Key: SAMZA-1523
> URL: https://issues.apache.org/jira/browse/SAMZA-1523
> Project: Samza
>  Issue Type: Bug
>Reporter: Navina Ramesh
>Assignee: Navina Ramesh
>Priority: Major
>
> We want to remove expired entries of the processors from the Azure Table when 
> the processor is shutting down. Azure Table service uses optimistic locking 
> by default. Hence, when the coordinator thread is cleaning up during 
> shutdown, it is possible for the heartbeat thread to update that entry as 
> well. This causes a failure in cleanup and throws exceptions in the log. 
> Obviously, it also fails to clear the entries :)
> `
> 2017-11-30 15:23:32.804 [JMVersionUpgradeScheduler-0] AzureJobCoordinator 
> [INFO] pid=05133d0a-dd85-4178-a97c-2c98dc617308new version 5 of the job model 
> got confirmed
> 2017-11-30 15:23:32.833 [HeartbeatScheduler-0] HeartbeatScheduler [INFO] 
> Updating heartbeat for processor ID: 05133d0a-dd85-4178-a97c-2c98dc617308 and 
> job model version: 4
> 2017-11-30 15:23:32.905 [JMVersionUpgradeScheduler-0] TableUtils [ERROR] 
> Azure storage exception while deleting processor entity with job model 
> version: 4and pid: 05133d0a-dd85-4178-a97c-2c98dc617308
> com.microsoft.azure.storage.table.TableServiceException: Precondition Failed
>   at 
> com.microsoft.azure.storage.table.TableServiceException.generateTableServiceException(TableServiceException.java:52)
>   at 
> com.microsoft.azure.storage.table.TableOperation$1.preProcessResponse(TableOperation.java:319)
>   at 
> com.microsoft.azure.storage.table.TableOperation$1.preProcessResponse(TableOperation.java:299)
>   at 
> com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:139)
>   at 
> com.microsoft.azure.storage.table.TableOperation.performDelete(TableOperation.java:281)
>   at 
> com.microsoft.azure.storage.table.TableOperation.execute(TableOperation.java:685)
>   at 
> com.microsoft.azure.storage.table.CloudTable.execute(CloudTable.java:529)
>   at 
> com.microsoft.azure.storage.table.CloudTable.execute(CloudTable.java:496)
>   at 
> org.apache.samza.util.TableUtils.deleteProcessorEntity(TableUtils.java:157)
>   at 
> org.apache.samza.coordinator.AzureJobCoordinator.onNewJobModelConfirmed(AzureJobCoordinator.java:448)
>   at 
> org.apache.samza.coordinator.AzureJobCoordinator.onNewJobModelAvailable(AzureJobCoordinator.java:419)
>   at 
> org.apache.samza.coordinator.AzureJobCoordinator.lambda$createJMVersionUpgradeListener$3(AzureJobCoordinator.java:248)
>   at 
> org.apache.samza.coordinator.scheduler.JMVersionUpgradeScheduler.lambda$scheduleTask$0(JMVersionUpgradeScheduler.java:81)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-11-30 15:23:32.906 [JMVersionUpgradeScheduler-0] AzureJobCoordinator 
> [ERROR] Exception in Job Model Version Upgrade Scheduler. Stopping the 
> processor...
> `
> We should disable optimistic locking during the cleanup phase of shutdown.  
> Ideal solution is to perhaps have more control over the various schedulers. 
> That is beyond the scope of this JIRA though :) 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1556) Samza SQL: Add support for multi level sources in SQL query

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329126#comment-16329126
 ] 

ASF GitHub Bot commented on SAMZA-1556:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/405


> Samza SQL: Add support for multi level sources in SQL query
> ---
>
> Key: SAMZA-1556
> URL: https://issues.apache.org/jira/browse/SAMZA-1556
> Project: Samza
>  Issue Type: Bug
>  Components: sql
>Reporter: Srinivasulu Punuru
>Priority: Major
>
> Right now Samza SQL supports queries with just two levels i.e. select * from 
> foo.bar. But there can be sources that are identified though multiple levels. 
> for e.g. select * from kafka.clusterName.topicName. We should be able to 
> support queries with such sources.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1500) Add metrics for RocksDB state store memory usage

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329075#comment-16329075
 ] 

ASF GitHub Bot commented on SAMZA-1500:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/404


> Add metrics for RocksDB state store memory usage
> 
>
> Key: SAMZA-1500
> URL: https://issues.apache.org/jira/browse/SAMZA-1500
> Project: Samza
>  Issue Type: Task
>Reporter: Prateek Maheshwari
>Assignee: Prateek Maheshwari
>Priority: Major
> Fix For: 0.15.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SAMZA-1556) Samza SQL: Add support for multi level sources in SQL query

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323376#comment-16323376
 ] 

ASF GitHub Bot commented on SAMZA-1556:
---

GitHub user srinipunuru opened a pull request:

https://github.com/apache/samza/pull/405

SAMZA-1556: Adding support for multi level sources in queries

Right now Samza SQL supports queries with just two levels i.e. `select * 
from foo.bar`. But there can be sources that are identified though multiple 
levels. for e.g. `select * from kafka.clusterName.topicName`. 

This change adds the support for sql queries with sources that have more 
than two levels.  

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/srinipunuru/samza multi-level.1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/405.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #405


commit bb667c7de66e07b1fb26707f2cc62137a25caee9
Author: Srinivasulu Punuru 
Date:   2018-01-12T01:19:10Z

SAMZA-1556: Adding support for multi level sources in queries




> Samza SQL: Add support for multi level sources in SQL query
> ---
>
> Key: SAMZA-1556
> URL: https://issues.apache.org/jira/browse/SAMZA-1556
> Project: Samza
>  Issue Type: Bug
>  Components: sql
>Reporter: Srinivasulu Punuru
>
> Right now Samza SQL supports queries with just two levels i.e. select * from 
> foo.bar. But there can be sources that are identified though multiple levels. 
> for e.g. select * from kafka.clusterName.topicName. We should be able to 
> support queries with such sources.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SAMZA-1500) Add metrics for RocksDB state store memory usage

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323067#comment-16323067
 ] 

ASF GitHub Bot commented on SAMZA-1500:
---

GitHub user prateekm opened a pull request:

https://github.com/apache/samza/pull/404

SAMZA-1500: Added metrics for RocksDB state store memory usage



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/prateekm/samza rocksdb-memory

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/404.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #404


commit 873598d8e4b9a8c96c9d21b837f754825e6a85f9
Author: Prateek Maheshwari 
Date:   2018-01-11T22:11:23Z

SAMZA-1500: Added metrics for RocksDB state store memory usage




> Add metrics for RocksDB state store memory usage
> 
>
> Key: SAMZA-1500
> URL: https://issues.apache.org/jira/browse/SAMZA-1500
> Project: Samza
>  Issue Type: Task
>Reporter: Prateek Maheshwari
>Assignee: Prateek Maheshwari
> Fix For: 0.15.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SAMZA-1535) Support UDFS in where clause

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320894#comment-16320894
 ] 

ASF GitHub Bot commented on SAMZA-1535:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/386


> Support UDFS in where clause
> 
>
> Key: SAMZA-1535
> URL: https://issues.apache.org/jira/browse/SAMZA-1535
> Project: Samza
>  Issue Type: Bug
>Reporter: Srinivasulu Punuru
> Fix For: 0.15.0
>
>
> The existing version of the udf implementation doesn't seem to supported in 
> the where clauses because the Type of the object returned is "ANY" and when 
> you do a
> select * from kafka.topic where regexMatch('.*foo', Name) it fails in the 
> query validation, because calcite doesn't know the type of regexMatch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SAMZA-1530) Bump up Kafka dependency to 0.11

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320841#comment-16320841
 ] 

ASF GitHub Bot commented on SAMZA-1530:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/395


> Bump up Kafka dependency to 0.11
> 
>
> Key: SAMZA-1530
> URL: https://issues.apache.org/jira/browse/SAMZA-1530
> Project: Samza
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
> Fix For: 0.15.0
>
>
> See http://kafka.apache.org/0110/documentation.html#upgrade_1100_notable for 
> notable changes in Kafka 0.11.
> One of the important feature in 0.11 is the ability to delete records on 
> demand. Samza can use this feature to save cost by deleting data from 
> intermediate Samza topics. See SAMZA-1478 for more detail.
> We need to do performance test and capability test before we bumping up the 
> Kafka version in the Samza master branch



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SAMZA-1553) Add log4j for latest Kafka build

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318916#comment-16318916
 ] 

ASF GitHub Bot commented on SAMZA-1553:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/402


> Add log4j for latest Kafka build
> 
>
> Key: SAMZA-1553
> URL: https://issues.apache.org/jira/browse/SAMZA-1553
> Project: Samza
>  Issue Type: Bug
>Reporter: Bharath Kumarasubramanian
>Assignee: Xinyu Liu
>
> Samza doesn't compile with the new Kafka build:
> {code}
>  [ant:scalac] error: bad symbolic reference. A signature in Logging.class 
> refers to term log4j
> [ant:scalac] in package org.apache which is not available.
> [ant:scalac] It may be completely missing from the current classpath, or the 
> version on
> [ant:scalac] the classpath might be incompatible with the version used when 
> compiling Logging.class.
> {code}
> Seems some of the kafka class used in Samza now has a compile time dependency 
> in log4j. Need to add it to build.gradle.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SAMZA-1553) Add log4j for latest Kafka build

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318835#comment-16318835
 ] 

ASF GitHub Bot commented on SAMZA-1553:
---

GitHub user xinyuiscool opened a pull request:

https://github.com/apache/samza/pull/402

SAMZA-1553: Add log4j for latest Kafka build

Add it so Samza compiles with the latest kafka.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xinyuiscool/samza SAMZA-1553

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/402.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #402


commit 1fe4d5c836f1b19c22ae8dd5af99a6060964eef4
Author: xiliu 
Date:   2018-01-09T18:02:54Z

SAMZA-1553: Add log4j for latest Kafka build




> Add log4j for latest Kafka build
> 
>
> Key: SAMZA-1553
> URL: https://issues.apache.org/jira/browse/SAMZA-1553
> Project: Samza
>  Issue Type: Bug
>Reporter: Bharath Kumarasubramanian
>Assignee: Xinyu Liu
>
> Samza doesn't compile with the new Kafka build:
> {code}
>  [ant:scalac] error: bad symbolic reference. A signature in Logging.class 
> refers to term log4j
> [ant:scalac] in package org.apache which is not available.
> [ant:scalac] It may be completely missing from the current classpath, or the 
> version on
> [ant:scalac] the classpath might be incompatible with the version used when 
> compiling Logging.class.
> {code}
> Seems some of the kafka class used in Samza now has a compile time dependency 
> in log4j. Need to add it to build.gradle.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SAMZA-1530) Bump up Kafka dependency to 0.11

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316761#comment-16316761
 ] 

ASF GitHub Bot commented on SAMZA-1530:
---

GitHub user lindong28 reopened a pull request:

https://github.com/apache/samza/pull/395

SAMZA-1530; Bump up Kafka dependency to 0.11



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lindong28/samza SAMZA-1530

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/395.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #395


commit ff02efd07fcfbccde224b9be254621c88af26d0d
Author: Dong Lin 
Date:   2018-01-02T18:37:36Z

SAMZA-1530; Bump up Kafka dependency to 0.11

commit 0448e4af331406db8d74bb9a49c24c256858ee0e
Author: Dong Lin 
Date:   2018-01-03T23:13:42Z

Address comments




> Bump up Kafka dependency to 0.11
> 
>
> Key: SAMZA-1530
> URL: https://issues.apache.org/jira/browse/SAMZA-1530
> Project: Samza
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
> Fix For: 0.15.0
>
>
> See http://kafka.apache.org/0110/documentation.html#upgrade_1100_notable for 
> notable changes in Kafka 0.11.
> One of the important feature in 0.11 is the ability to delete records on 
> demand. Samza can use this feature to save cost by deleting data from 
> intermediate Samza topics. See SAMZA-1478 for more detail.
> We need to do performance test and capability test before we bumping up the 
> Kafka version in the Samza master branch



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SAMZA-1530) Bump up Kafka dependency to 0.11

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316720#comment-16316720
 ] 

ASF GitHub Bot commented on SAMZA-1530:
---

Github user lindong28 closed the pull request at:

https://github.com/apache/samza/pull/395


> Bump up Kafka dependency to 0.11
> 
>
> Key: SAMZA-1530
> URL: https://issues.apache.org/jira/browse/SAMZA-1530
> Project: Samza
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
> Fix For: 0.15.0
>
>
> See http://kafka.apache.org/0110/documentation.html#upgrade_1100_notable for 
> notable changes in Kafka 0.11.
> One of the important feature in 0.11 is the ability to delete records on 
> demand. Samza can use this feature to save cost by deleting data from 
> intermediate Samza topics. See SAMZA-1478 for more detail.
> We need to do performance test and capability test before we bumping up the 
> Kafka version in the Samza master branch



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SAMZA-1552) Host affinity improvements - Improve matching of hosts to allocated resources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314287#comment-16314287
 ] 

ASF GitHub Bot commented on SAMZA-1552:
---

GitHub user vjagadish1989 opened a pull request:

https://github.com/apache/samza/pull/401

SAMZA-1552: Host affinity improvements - Improve matching of hosts to 
allocated resources



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vjagadish1989/samza host-affinity-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/401.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #401


commit 73c230989f16f9e2ffcaefbe5b9610ef6bc818a6
Author: Jagadish 
Date:   2018-01-05T08:46:00Z

Host affinity fixes




> Host affinity improvements - Improve matching of hosts to allocated resources
> -
>
> Key: SAMZA-1552
> URL: https://issues.apache.org/jira/browse/SAMZA-1552
> Project: Samza
>  Issue Type: Bug
>Reporter: Abhishek Shivanna
>Assignee: Jagadish
>
> Kudos to [~abkshvn] for observing this!
> We have observed host-affinity not being honored for some containers in very 
> large jobs. When Yarn allocates more resources than what Samza requested on a 
> specific host, the extra resources are added to a spare-pool called the 
> "ANY_HOST Buffer". Later, when there is a resource request for the same host 
> from Samza and Yarn does not return resources, we don't leverage the 
> spare-pool of previously returned resources in that host. 
> This problem is specially pronounced in clusters that are heavily loaded in 
> cpu, and memory where allocations need to satisfy both cpu and memory 
> requirements of available hosts (Often, hosts have cpu but not memory or 
> vice-versa). If there are a lot of container failures on a particular host in 
> the midst of allocation, it further aggravates this problem.
> The fix is as follows:
> Check if there are available containers in the buffer corresponding to our 
> preferred host. If not, we should also scan the ANY-HOST buffer for matched 
> containers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SAMZA-1550) Doc for 0.14.0 release

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310505#comment-16310505
 ] 

ASF GitHub Bot commented on SAMZA-1550:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/400


> Doc for 0.14.0 release
> --
>
> Key: SAMZA-1550
> URL: https://issues.apache.org/jira/browse/SAMZA-1550
> Project: Samza
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Xinyu Liu
>Assignee: Xinyu Liu
> Fix For: 0.14.0
>
>
> Update docs for 0.14.0 release.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SAMZA-1408) update the doc for 0.13.1 release

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310374#comment-16310374
 ] 

ASF GitHub Bot commented on SAMZA-1408:
---

Github user fredji97 closed the pull request at:

https://github.com/apache/samza/pull/286


> update the doc for 0.13.1 release
> -
>
> Key: SAMZA-1408
> URL: https://issues.apache.org/jira/browse/SAMZA-1408
> Project: Samza
>  Issue Type: Task
>Reporter: Fred Ji
>Assignee: Fred Ji
>
> This is required by following docs/README.md for the release. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SAMZA-1550) Doc for 0.14.0 release

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310194#comment-16310194
 ] 

ASF GitHub Bot commented on SAMZA-1550:
---

GitHub user xinyuiscool opened a pull request:

https://github.com/apache/samza/pull/400

SAMZA-1550: Update master to use 0.14.1-SNAPSHOT version

Update master to use 0.14.1-SNAPSHOT version.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xinyuiscool/samza SAMZA-1550-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/400.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #400


commit 543b0489849804cc1aec792c2c389fbc83337edb
Author: xiliu 
Date:   2018-01-03T19:49:16Z

SAMZA-1550: Update master to use 0.14.1-SNAPSHOT version




> Doc for 0.14.0 release
> --
>
> Key: SAMZA-1550
> URL: https://issues.apache.org/jira/browse/SAMZA-1550
> Project: Samza
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Xinyu Liu
>Assignee: Xinyu Liu
> Fix For: 0.14.0
>
>
> Update docs for 0.14.0 release.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SAMZA-1550) Doc for 0.14.0 release

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310160#comment-16310160
 ] 

ASF GitHub Bot commented on SAMZA-1550:
---

Github user xinyuiscool closed the pull request at:

https://github.com/apache/samza/pull/399


> Doc for 0.14.0 release
> --
>
> Key: SAMZA-1550
> URL: https://issues.apache.org/jira/browse/SAMZA-1550
> Project: Samza
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Xinyu Liu
>Assignee: Xinyu Liu
> Fix For: 0.14.0
>
>
> Update docs for 0.14.0 release.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SAMZA-1550) Doc for 0.14.0 release

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310150#comment-16310150
 ] 

ASF GitHub Bot commented on SAMZA-1550:
---

GitHub user xinyuiscool opened a pull request:

https://github.com/apache/samza/pull/399

[SAMZA-1550]: replace snapshot with release version in 0.14.0 branch

Prepare the doc for the 0.14.0 branch.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xinyuiscool/samza SAMZA-1550-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/399.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #399


commit 645603dd427bd7b86295c6d337d494f5ffd5f786
Author: xiliu 
Date:   2018-01-03T19:23:49Z

[SAMZA-1550]: replace snapshot with release version in 0.14.0 branch




> Doc for 0.14.0 release
> --
>
> Key: SAMZA-1550
> URL: https://issues.apache.org/jira/browse/SAMZA-1550
> Project: Samza
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Xinyu Liu
>Assignee: Xinyu Liu
> Fix For: 0.14.0
>
>
> Update docs for 0.14.0 release.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SAMZA-1550) Doc for 0.14.0 release

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16309996#comment-16309996
 ] 

ASF GitHub Bot commented on SAMZA-1550:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/396


> Doc for 0.14.0 release
> --
>
> Key: SAMZA-1550
> URL: https://issues.apache.org/jira/browse/SAMZA-1550
> Project: Samza
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Xinyu Liu
>Assignee: Xinyu Liu
> Fix For: 0.14.0
>
>
> Update docs for 0.14.0 release.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SAMZA-1542) migrate config classes in samza-core to use composition over inherence

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308939#comment-16308939
 ] 

ASF GitHub Bot commented on SAMZA-1542:
---

GitHub user fredji97 opened a pull request:

https://github.com/apache/samza/pull/398

SAMZA-1542 refactor config classes in samza-core java to hide non-necessary 
interfaces

With previous inheritance, it exposed everything from Config class, which 
is a bad practice and is error prone. We are refactoring it to use composition 
over inheritance make it more robust. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/fredji97/samza samzaCoreJavaConfig

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/398.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #398


commit 34dd7f40f044b7c219e402dbc4575579d0c42643
Author: Fred Ji 
Date:   2018-01-03T00:51:47Z

SAMZA-1542 refactor config classes in samza-core java to hide non-necessary 
interfaces




> migrate config classes in samza-core to use composition over inherence
> --
>
> Key: SAMZA-1542
> URL: https://issues.apache.org/jira/browse/SAMZA-1542
> Project: Samza
>  Issue Type: Improvement
>Reporter: Fred Ji
>Assignee: Fred Ji
> Fix For: 0.15.0
>
>
> This is one of the child tasks from 
> https://issues.apache.org/jira/browse/SAMZA-1160



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SAMZA-1548) Add start() and stop() to SystemAdmin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308827#comment-16308827
 ] 

ASF GitHub Bot commented on SAMZA-1548:
---

GitHub user lindong28 opened a pull request:

https://github.com/apache/samza/pull/397

SAMZA-1548; Add start() and stop() to SystemAdmin



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lindong28/samza SAMZA-1548

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/397.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #397


commit aae4247aef2ab5eee79c31ea09c3a0f1f0b94e53
Author: Dong Lin 
Date:   2018-01-02T19:42:17Z

SAMZA-1548; Add start() and stop() to SystemAdmin




> Add start() and stop() to SystemAdmin
> -
>
> Key: SAMZA-1548
> URL: https://issues.apache.org/jira/browse/SAMZA-1548
> Project: Samza
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SAMZA-1550) Doc for 0.14.0 release

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308652#comment-16308652
 ] 

ASF GitHub Bot commented on SAMZA-1550:
---

GitHub user xinyuiscool opened a pull request:

https://github.com/apache/samza/pull/396

SAMZA-1550: Doc for 0.14.0 release

Docs update for both master and 0.14.0 branch.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xinyuiscool/samza SAMZA-1550

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/396.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #396


commit f1088dcb70b793484fc56988aa2617b1fddb2a7b
Author: xiliu 
Date:   2018-01-02T20:19:03Z

SAMZA-1550: Doc for 0.14.0 release




> Doc for 0.14.0 release
> --
>
> Key: SAMZA-1550
> URL: https://issues.apache.org/jira/browse/SAMZA-1550
> Project: Samza
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Xinyu Liu
>Assignee: Xinyu Liu
> Fix For: 0.14.0
>
>
> Update docs for 0.14.0 release.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SAMZA-1528) Implement async NodeManager interactions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308646#comment-16308646
 ] 

ASF GitHub Bot commented on SAMZA-1528:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/380


> Implement async NodeManager interactions
> 
>
> Key: SAMZA-1528
> URL: https://issues.apache.org/jira/browse/SAMZA-1528
> Project: Samza
>  Issue Type: Bug
>Reporter: Jagadish
>Assignee: Jagadish
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SAMZA-1530) Bump up Kafka dependency to 0.11

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308514#comment-16308514
 ] 

ASF GitHub Bot commented on SAMZA-1530:
---

GitHub user lindong28 opened a pull request:

https://github.com/apache/samza/pull/395

SAMZA-1530; Bump up Kafka dependency to 0.11



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lindong28/samza SAMZA-1530

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/395.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #395


commit f264d1dd45cbf6d5395964e3a42c0b629d4ed09b
Author: Dong Lin 
Date:   2018-01-02T18:37:36Z

SAMZA-1530; Bump up Kafka dependency to 0.11




> Bump up Kafka dependency to 0.11
> 
>
> Key: SAMZA-1530
> URL: https://issues.apache.org/jira/browse/SAMZA-1530
> Project: Samza
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
>
> See http://kafka.apache.org/0110/documentation.html#upgrade_1100_notable for 
> notable changes in Kafka 0.11.
> One of the important feature in 0.11 is the ability to delete records on 
> demand. Samza can use this feature to save cost by deleting data from 
> intermediate Samza topics. See SAMZA-1478 for more detail.
> We need to do performance test and capability test before we bumping up the 
> Kafka version in the Samza master branch



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SAMZA-1547) Fix default value for the grouper-factory configuration in KafkaCheckpointManager

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301990#comment-16301990
 ] 

ASF GitHub Bot commented on SAMZA-1547:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/394


> Fix default value for the grouper-factory configuration in 
> KafkaCheckpointManager
> -
>
> Key: SAMZA-1547
> URL: https://issues.apache.org/jira/browse/SAMZA-1547
> Project: Samza
>  Issue Type: Bug
>Reporter: Jagadish
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SAMZA-1547) Fix default value for the grouper-factory configuration in KafkaCheckpointManager

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301951#comment-16301951
 ] 

ASF GitHub Bot commented on SAMZA-1547:
---

GitHub user vjagadish1989 opened a pull request:

https://github.com/apache/samza/pull/394

SAMZA-1547: Parse default value grouper-factory config in KafkaCheckpointMgr

- Additionally, updated unit-tests.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vjagadish1989/samza kcm-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/394.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #394


commit 3a08518509c311397e23692e9bb1bad037c96a68
Author: Jagadish 
Date:   2017-12-22T20:49:50Z

SAMZA-1547: Fix default handling of grouper-factory configuration in 
KafkaCheckpointManager




> Fix default value for the grouper-factory configuration in 
> KafkaCheckpointManager
> -
>
> Key: SAMZA-1547
> URL: https://issues.apache.org/jira/browse/SAMZA-1547
> Project: Samza
>  Issue Type: Bug
>Reporter: Jagadish
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SAMZA-1539) KafkaProducer potential hang on close() when task.drop.producer.errors==true

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301783#comment-16301783
 ] 

ASF GitHub Bot commented on SAMZA-1539:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/390


> KafkaProducer potential hang on close() when task.drop.producer.errors==true
> 
>
> Key: SAMZA-1539
> URL: https://issues.apache.org/jira/browse/SAMZA-1539
> Project: Samza
>  Issue Type: Bug
>Reporter: Jake Maes
>Assignee: Jake Maes
> Fix For: 0.15.0
>
>
> The issue is caused by 2 bad behaviors:
> 1. KafkaProducer.close() does an unbounded thread.join() even when a force 
> close (timeout == 0 ms) is specified. 
> 2. KafkaSystemProducer.handleSendException() has a synchronized block to 
> recreate the producer when task.drop.producer.errors==true. When the IO 
> thread is waiting on the synchronized block, it prevents the KafkaProducer 
> from joining the IO thread.
> Fixing either one will fix the issue, but we will close this ticket as soon 
> as the Samza side is tested and committed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SAMZA-1541) migrate config classes in samza-yarn to use composition over inherence

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16300741#comment-16300741
 ] 

ASF GitHub Bot commented on SAMZA-1541:
---

GitHub user fredji97 opened a pull request:

https://github.com/apache/samza/pull/392

SAMZA-1541 migrate config classes in samza-yarn to use composition over 
inherence

./gradlew clean check successfully.
The reason for doing this is described in 
https://issues.apache.org/jira/browse/SAMZA-1160. 
I am going to migrate for each directory and this PR is for samza-yarn. 
Besides the refactoring, two config class files 
(LocalizerResourceConfig.java and FileSystemImplConfig.java) are also moved 
from org.apache.samza.job.yarn to org.apache.samza.config. 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/fredji97/samza samzaYarnConfig

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/392.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #392


commit b3a64c9384c54a206c60505a04c3eed41a6a562b
Author: Fred Ji 
Date:   2017-12-21T23:49:04Z

SAMZA-1541 migrate config classes in samza-yarn to use composition over 
inherence




> migrate config classes in samza-yarn to use composition over inherence
> --
>
> Key: SAMZA-1541
> URL: https://issues.apache.org/jira/browse/SAMZA-1541
> Project: Samza
>  Issue Type: Improvement
>Reporter: Fred Ji
>Assignee: Fred Ji
> Fix For: 0.15.0
>
>
> This is one of the child tasks from 
> https://issues.apache.org/jira/browse/SAMZA-1160



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SAMZA-1537) StreamAppender can deadlock due to locks held by Kafka and Log4j

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16300482#comment-16300482
 ] 

ASF GitHub Bot commented on SAMZA-1537:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/388


> StreamAppender can deadlock due to locks held by Kafka and Log4j
> 
>
> Key: SAMZA-1537
> URL: https://issues.apache.org/jira/browse/SAMZA-1537
> Project: Samza
>  Issue Type: Bug
>Reporter: Jake Maes
>Assignee: Jake Maes
>
> The thread dumps of the 2 offending threads are below, but the basics are: 
> 1. AppInfoParser in kafka uses static synchronized methods
> 2. Log4j synchronizes per Category
> So if the StreamAppender tries create a new KafkaProducer, which calls the 
> static sync AppInfoParser thread, which then tries to log to the same Category
> {noFormat}
> "kafka-producer-network-thread | kafka_producer-samza_xxx_yyy-i001" daemon 
> prio=5 tid=23 BLOCKED
>   at 
> org.apache.kafka.common.utils.AppInfoParser.registerAppInfo(AppInfoParser.java:57)
>  Local Variable: java.lang.String#326563
>  Local Variable: java.lang.String#329864
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:336)
>  Local Variable: 
> org.apache.kafka.common.config.AbstractConfig$RecordingMap#12
>  Local Variable: java.util.ArrayList#265184
>  Local Variable: org.apache.kafka.common.metrics.MetricConfig#9
>  Local Variable: java.util.LinkedHashMap#991
>  Local Variable: 
> org.apache.kafka.common.internals.ClusterResourceListeners#9
>  Local Variable: java.util.ArrayList#265353
>  Local Variable: org.apache.kafka.clients.NetworkClient#9
>  Local Variable: org.apache.kafka.common.network.SslChannelBuilder#9
>  Local Variable: java.util.ArrayList#265374
>  Local Variable: org.apache.kafka.clients.producer.ProducerConfig#3
>  Local Variable: java.lang.String#309971
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:182)
>  Local Variable: 
> org.apache.kafka.common.config.AbstractConfig$RecordingMap#11
>  Local Variable: org.apache.kafka.clients.producer.KafkaProducer#3
>  Local Variable: 
> org.apache.kafka.common.serialization.ByteArraySerializer#7
>  Local Variable: 
> org.apache.kafka.common.serialization.ByteArraySerializer#8
>   at 
> com.linkedin.kafka.liclients.producer.LiKafkaProducerImpl.(LiKafkaProducerImpl.java:159)
>  Local Variable: 
> com.linkedin.kafka.liclients.producer.LiKafkaProducerConfig#2
>   at 
> com.linkedin.kafka.liclients.producer.LiKafkaProducerImpl.(LiKafkaProducerImpl.java:137)
>  Local Variable: 
> org.apache.kafka.common.serialization.ByteArraySerializer#10
>  Local Variable: 
> org.apache.kafka.common.serialization.ByteArraySerializer#9
>  Local Variable: java.util.Properties#38
>  Local Variable: 
> com.linkedin.samza.system.kafka.serializers.NoOpSegmentSerializer#3
>  Local Variable: com.linkedin.kafka.liclients.auditing.NoOpAuditor#9
>  Local Variable: 
> com.linkedin.kafka.liclients.producer.LiKafkaProducerImpl#3
>   at 
> com.linkedin.samza.system.kafka.SamzaRawLiKafkaSystemProducer.createLiKafkaProducer(SamzaRawLiKafkaSystemProducer.java:84)
>   at 
> com.linkedin.samza.system.kafka.SamzaRawLiKafkaSystemProducer.handleSendException(SamzaRawLiKafkaSystemProducer.java:224)
>   at 
> com.linkedin.samza.system.kafka.SamzaRawLiKafkaSystemProducer.send(SamzaRawLiKafkaSystemProducer.java:167)
>  Local Variable: java.lang.String#326561
>  Local Variable: java.lang.IllegalStateException#2
>  Local Variable: java.lang.String#330077
>  Local Variable: org.apache.samza.system.SystemProducerException#4
>  Local Variable: java.lang.Integer#15116
>   at 
> org.apache.samza.logging.log4j.StreamAppender.append(StreamAppender.java:115)
>   at 
> com.linkedin.atc.log4j.SafeStreamAppender.streamAppend(SafeStreamAppender.java:32)
>   at 
> com.linkedin.atc.log4j.SafeStreamAppender.append(SafeStreamAppender.java:23)
>  Local Variable: com.linkedin.atc.log4j.SafeStreamAppender#1
>   at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
>   at 
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
>  Local Variable: org.apache.log4j.helpers.AppenderAttachableImpl#1
>   at org.apache.log4j.Category.callAppenders(Category.java:206)
>  Local Variable: org.apache.log4j.spi.LoggingEvent#24
>  Local Variable: org.apache.log4j.Logger#4
>   at org.apache.log4j.Category.forcedLog(Category.java:391)
>   at 

[jira] [Commented] (SAMZA-1539) KafkaProducer potential hang on close() when task.drop.producer.errors==true

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299030#comment-16299030
 ] 

ASF GitHub Bot commented on SAMZA-1539:
---

GitHub user jmakes opened a pull request:

https://github.com/apache/samza/pull/390

SAMZA-1539: KafkaProducer potential hang on close() when task.drop.pr…

…oducer.errors==true

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jmakes/samza samza-1539

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/390.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #390


commit 87a39fc076dde94574590dff6ae7fd070f012a64
Author: Jacob Maes 
Date:   2017-12-20T20:21:59Z

SAMZA-1539: KafkaProducer potential hang on close() when 
task.drop.producer.errors==true




> KafkaProducer potential hang on close() when task.drop.producer.errors==true
> 
>
> Key: SAMZA-1539
> URL: https://issues.apache.org/jira/browse/SAMZA-1539
> Project: Samza
>  Issue Type: Bug
>Reporter: Jake Maes
>Assignee: Jake Maes
>
> The issue is caused by 2 bad behaviors:
> 1. KafkaProducer.close() does an unbounded thread.join() even when a force 
> close (timeout == 0 ms) is specified. 
> 2. KafkaSystemProducer.handleSendException() has a synchronized block to 
> recreate the producer when task.drop.producer.errors==true. When the IO 
> thread is waiting on the synchronized block, it prevents the KafkaProducer 
> from joining the IO thread.
> Fixing either one will fix the issue, but we will close this ticket as soon 
> as the Samza side is tested and committed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SAMZA-1356) Improve monitoring for state restore

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297006#comment-16297006
 ] 

ASF GitHub Bot commented on SAMZA-1356:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/241


> Improve monitoring for state restore
> 
>
> Key: SAMZA-1356
> URL: https://issues.apache.org/jira/browse/SAMZA-1356
> Project: Samza
>  Issue Type: Bug
>Reporter: Jake Maes
>Assignee: Jake Maes
> Fix For: 0.14.0
>
>
> There are a couple problems that can affect our ability to troubleshoot state 
> restore from changelog.
> 1. KeyValueStorageEngine logs a message for every 1M messages restored, but 
> it doesn't print anything for smaller stores. We should add a message to 
> report the final number of entries restored.
> 2. While the "restore-time" metric is a gauge, the 
> KeyValueStorageEngineMetrics "messages-restored" and "messages-bytes" are 
> both counters, and counters are often graphed in terms of deltas so the value 
> disappears after one data point. Since these values only matter for the 
> beginning of the job, we should switch them to gauges so the value is 
> retained for later monitoring. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SAMZA-1538) Flaky test: TestStreamProcessor testStreamProcessor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SAMZA-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295952#comment-16295952
 ] 

ASF GitHub Bot commented on SAMZA-1538:
---

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/389


> Flaky test: TestStreamProcessor testStreamProcessor
> ---
>
> Key: SAMZA-1538
> URL: https://issues.apache.org/jira/browse/SAMZA-1538
> Project: Samza
>  Issue Type: Bug
>Reporter: Prateek Maheshwari
>
> Also, testStreamProcessorWithAsyncStreamTaskFactory and 
> testStreamProcessorWithStreamTaskFactory.
> Fails frequently on Jenkins:
> https://travis-ci.org/apache/samza/builds/308557207utm_source=github_status_medium=notification
> https://travis-ci.org/apache/samza/builds/308755404utm_source=github_status_medium=notification
> https://travis-ci.org/apache/samza/builds/309139716utm_source=github_status_medium=notification



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)