Re: Review Request 50174: SAMZA-977: User doc for samza multithreading

2016-07-27 Thread Xinyu Liu
. True, my doc was overly simplified. I added more details. Please take a look. - Xinyu ------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50174/#review142985 --- On July 27, 2016, 11:05 p.m., X

Re: Review Request 50174: SAMZA-977: User doc for samza multithreading

2016-07-27 Thread Xinyu Liu
/diff/ Testing --- Test the web pages locally. Thanks, Xinyu Liu

Review Request 50527: SAMZA-970: fix integration tests

2016-07-27 Thread Xinyu Liu
/samza_job_yarn_deployer.py 38635ca5899c43fb61d6b4042e8543f0508fd41b Diff: https://reviews.apache.org/r/50527/diff/ Testing --- run integration tests and now they work fine. Thanks, Xinyu Liu

Re: 0.10.1 Release

2016-07-27 Thread xinyu liu
Yes, it is undergoing. We were blocked by the integration tests ( https://issues.apache.org/jira/browse/SAMZA-970) and other issues for a while. Now we got a fix for the tests and I will submit a patch today. The rest of release should be quick. I will sync with Navina with an ETA. Sorry about the

Re: Review Request 50451: SAMZA-981: Set consistent Kafka clientId for a job instance

2016-07-26 Thread Xinyu Liu
-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemConsumer.scala ece0359d1a31e65113f4832dd3baddb788cecef4 Diff: https://reviews.apache.org/r/50451/diff/ Testing --- gradle build & tests Thanks, Xinyu Liu

Re: Review Request 50451: SAMZA-981: Set consistent Kafka clientId for a job instance

2016-07-26 Thread Xinyu Liu
onfig/TestKafkaConfig.scala c4a83f60cf049e6fd65f940b69674895d07ab871 samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemConsumer.scala ece0359d1a31e65113f4832dd3baddb788cecef4 Diff: https://reviews.apache.org/r/50451/diff/ Testing --- gradle build & tests Thanks, Xinyu Liu

Re: Review Request 50451: SAMZA-981: Set consistent Kafka clientId for a job instance

2016-07-26 Thread Xinyu Liu
-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemConsumer.scala ece0359d1a31e65113f4832dd3baddb788cecef4 Diff: https://reviews.apache.org/r/50451/diff/ Testing --- gradle build & tests Thanks, Xinyu Liu

Review Request 50451: SAMZA-981: Set consistent Kafka clientId for a job instance

2016-07-26 Thread Xinyu Liu
--- gradle build & tests Thanks, Xinyu Liu

Re: Review Request 50318: SAMZA-979: Remove KafkaCheckpointMigration

2016-07-26 Thread Xinyu Liu
To reply, visit: https://reviews.apache.org/r/50318/#review143446 ------- On July 25, 2016, 9:29 p.m., Xinyu Liu wrote: > > --- > This is an automatically generated e-mail.

Re: Review Request 50318: SAMZA-979: Remove KafkaCheckpointMigration

2016-07-25 Thread Xinyu Liu
/50318/diff/ Testing --- Tested by locally deployed jobs. Passed gradle tests. Thanks, Xinyu Liu

Review Request 50318: SAMZA-979: Fix for KafkaCheckpointMigration not registering source correctly

2016-07-21 Thread Xinyu Liu
5d2641ab1e54d49f7b983bc526762cfb50f2911b Diff: https://reviews.apache.org/r/50318/diff/ Testing --- Tested by locally deployed jobs. Passed gradle tests. Thanks, Xinyu Liu

Re: Review Request 50056: SAMZA-863: Multithreading changes

2016-07-19 Thread Xinyu Liu
--- Tested in yarn. Thanks, Xinyu Liu

Review Request 50174: SAMZA-977: User doc for samza multithreading

2016-07-18 Thread Xinyu Liu
116238312df7071747cbbc14bc9c46f558755195 docs/learn/documentation/versioned/jobs/configuration-table.html 54c52981c3055b398ee60af50eeaf2592ed0e64f Diff: https://reviews.apache.org/r/50174/diff/ Testing --- Test the web pages locally. Thanks, Xinyu Liu

Re: Review Request 50056: SAMZA-863: Multithreading changes

2016-07-18 Thread Xinyu Liu
--- Tested in yarn. Thanks, Xinyu Liu

Re: Review Request 50056: SAMZA-863: Multithreading changes

2016-07-18 Thread Xinyu Liu
/ Testing --- Tested in yarn. Thanks, Xinyu Liu

Re: Review Request 50056: Multithreading changes

2016-07-18 Thread Xinyu Liu
/TestKeyValueStores.scala fd4e7624afced0df104d4a90539149197b93808b Diff: https://reviews.apache.org/r/50056/diff/ Testing --- Tested in yarn. Thanks, Xinyu Liu

Re: Review Request 50143: Support the notion of 'end-of-stream' in Samza when consuming from finite sources

2016-07-18 Thread Xinyu Liu
is thread safe since we might have multiple threads for a task. - Xinyu Liu On July 18, 2016, 4:29 p.m., Jagadish Venkatraman wrote: > > --- > This is an automatically generated e-mail. To reply, visit: >

Re: Review Request 50082: SAMZA-973: Disk Quotas: clamp max delay, better measure processing time

2016-07-15 Thread Xinyu Liu
/samza/container/RunLoop.scala (line 79) <https://reviews.apache.org/r/50082/#comment208023> Shall we move this line in the process() function? - Xinyu Liu On July 15, 2016, 7:59 p.m., Chris Pettitt wrote: > > ---

Review Request 50056: Multithreading changes

2016-07-14 Thread Xinyu Liu
96eb5faf5c2a6f8b4271a5ed16e709015d78bab0 samza-test/src/test/scala/org/apache/samza/storage/kv/TestKeyValueStores.scala fd4e7624afced0df104d4a90539149197b93808b Diff: https://reviews.apache.org/r/50056/diff/ Testing --- Tested in yarn. Thanks, Xinyu Liu

Re: Review Request 48213: SAMZA-960: Make system producer thread safe

2016-07-13 Thread Xinyu Liu
/samza/system/kafka/TestKafkaSystemProducer.scala 8e32bba6ced090f0fc8d4e5176fe0788df36981d Diff: https://reviews.apache.org/r/48213/diff/ Testing --- Unit tests and local testing. Thanks, Xinyu Liu

Re: Review Request 48243: SAMZA-961: Async tasks and multithreading model

2016-07-12 Thread Xinyu Liu
-core/src/test/scala/org/apache/samza/system/TestSystemConsumers.scala 09da62e0f9a10f7c3683345a309c6278ff01fb4b Diff: https://reviews.apache.org/r/48243/diff/ Testing --- unit tests and local testing. Thanks, Xinyu Liu

Re: Review Request 49877: SAMZA-972: Holistic memory monitoring for SamzaContainer

2016-07-11 Thread Xinyu Liu
/samza/container/host/ProcfsBasedStatisticsMonitor.java (line 177) <https://reviews.apache.org/r/49877/#comment207134> Do you think we should catch all the throwable here to prevent the thread being killed? - Xinyu Liu On July 11, 2016, 6:57 a.m., Jagadish Venkatraman

Re: Review Request 48243: SAMZA-961: Async tasks and multithreading model

2016-07-11 Thread Xinyu Liu
/ Testing --- unit tests and local testing. Thanks, Xinyu Liu

Re: Review Request 48243: SAMZA-961: Async tasks and multithreading model

2016-06-15 Thread Xinyu Liu
omatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48243/#review137543 ------- On June 15, 2016, 11:41 p.m., Xinyu Liu wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https

Re: Review Request 48243: SAMZA-961: Async tasks and multithreading model

2016-06-15 Thread Xinyu Liu
same comment on the last pass. I don't understand how this is > supposed to work. Might be worth discussing this in person with Xinyu and > then either fixing the code or the doc as appropriate. Thanks both of you to catch this. I messed up the code during the task state refactoring. Now the lo

Re: Review Request 48243: SAMZA-961: Async tasks and multithreading model

2016-06-15 Thread Xinyu Liu
ystem/TestSystemConsumers.scala 09da62e0f9a10f7c3683345a309c6278ff01fb4b Diff: https://reviews.apache.org/r/48243/diff/ Testing --- unit tests and local testing. Thanks, Xinyu Liu

Re: Review Request 48243: SAMZA-961: Async tasks and multithreading model

2016-06-09 Thread Xinyu Liu
rc/main/scala/org/apache/samza/container/RunLoop.scala, lines > > 73-77 > > <https://reviews.apache.org/r/48243/diff/1/?file=1406265#file1406265line73> > > > > Same comment as for the async run loop. Fixed. As commented above. - Xinyu -

Re: Review Request 48243: SAMZA-961: Async tasks and multithreading model

2016-06-09 Thread Xinyu Liu
28e78 samza-core/src/test/scala/org/apache/samza/system/TestSystemConsumers.scala 09da62e0f9a10f7c3683345a309c6278ff01fb4b Diff: https://reviews.apache.org/r/48243/diff/ Testing --- unit tests and local testing. Thanks, Xinyu Liu

Re: Review Request 48182: SAMZA-958: Make store/cache thread safe

2016-06-08 Thread Xinyu Liu
2/?file=1406146#file1406146line109> > > > > nit: unused variable? Good catch! Removed it. - Xinyu --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48182/#

Re: Review Request 48182: SAMZA-958: Make store/cache thread safe

2016-06-08 Thread Xinyu Liu
To reply, visit: https://reviews.apache.org/r/48182/#review136335 ------- On June 9, 2016, 12:33 a.m., Xinyu Liu wrote: > > --- > This is an automatically generated e-mail.

Re: Review Request 48182: SAMZA-958: Make store/cache thread safe

2016-06-08 Thread Xinyu Liu
Unit tests and local deployment. Thanks, Xinyu Liu

Re: Review Request 48213: SAMZA-960: Make system producer thread safe

2016-06-08 Thread Xinyu Liu
- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48213/#review136654 --- On June 8, 2016, 11:53 p.m., Xinyu Liu wrote: > > ---

Re: Review Request 48213: SAMZA-960: Make system producer thread safe

2016-06-08 Thread Xinyu Liu
/org/apache/samza/system/kafka/TestKafkaSystemProducer.scala 8e32bba6ced090f0fc8d4e5176fe0788df36981d Diff: https://reviews.apache.org/r/48213/diff/ Testing --- Unit tests and local testing. Thanks, Xinyu Liu

Re: Review Request 48213: SAMZA-960: Make system producer thread safe

2016-06-07 Thread Xinyu Liu
04c9113fd6c3dd56c49ff46c8c1c0ff12f68e5e2 samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemProducer.scala 8e32bba6ced090f0fc8d4e5176fe0788df36981d Diff: https://reviews.apache.org/r/48213/diff/ Testing --- Unit tests and local testing. Thanks, Xinyu Liu

Re: Review Request 48213: SAMZA-960: Make system producer thread safe

2016-06-07 Thread Xinyu Liu
of volatile reads you need to make. Right, a simple null check should do it. Thanks for the suggestion. - Xinyu --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48213/#review136338 --- On June

Review Request 48243: SAMZA-961: Async tasks and multithreading model

2016-06-03 Thread Xinyu Liu
/scala/org/apache/samza/system/TestSystemConsumers.scala 09da62e0f9a10f7c3683345a309c6278ff01fb4b Diff: https://reviews.apache.org/r/48243/diff/ Testing --- unit tests and local testing. Thanks, Xinyu Liu

Re: Review Request 48213: SAMZA-960: Make system producer thread safe

2016-06-03 Thread Xinyu Liu
g the producer will get the error of closed producer, instead of NPE. - Xinyu --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48213/#review136097 ----------

Re: Review Request 48213: SAMZA-960: Make system producer thread safe

2016-06-03 Thread Xinyu Liu
.java 04c9113fd6c3dd56c49ff46c8c1c0ff12f68e5e2 samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemProducer.scala 8e32bba6ced090f0fc8d4e5176fe0788df36981d Diff: https://reviews.apache.org/r/48213/diff/ Testing --- Unit tests and local testing. Thanks, Xinyu Liu

Re: Review Request 48182: SAMZA-958: Make store/cache thread safe

2016-06-03 Thread Xinyu Liu
-- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48182/#review136075 ------- On June 3, 2016, 9:30 p.m., Xinyu Liu wrote: > > ---

Re: Review Request 48182: SAMZA-958: Make store/cache thread safe

2016-06-03 Thread Xinyu Liu
Unit tests and local deployment. Thanks, Xinyu Liu

Review Request 48213: SAMZA-960: Make system producer thread safe

2016-06-03 Thread Xinyu Liu
/TestKafkaSystemProducer.scala 8e32bba6ced090f0fc8d4e5176fe0788df36981d Diff: https://reviews.apache.org/r/48213/diff/ Testing --- Unit tests and local testing. Thanks, Xinyu Liu

Review Request 48182: SAMZA-958: Make store/cache thread safe

2016-06-02 Thread Xinyu Liu
/scala/org/apache/samza/storage/kv/TestKeyValueStores.scala 23f8a1a6bee8ef38e0640a4e90778e53d982deeb Diff: https://reviews.apache.org/r/48182/diff/ Testing --- Unit tests and local deployment. Thanks, Xinyu Liu

Re: [DISCUSS] Re-thinking the Samza Job Coordinator

2016-03-02 Thread xinyu liu
This is awesome! Samza will be able to run standalone or on different kinds of clusters, which can serve a wide range of applications. Glad to see this is happening. Thanks, Xinyu On Tue, Mar 1, 2016 at 8:48 AM, Jagadish Venkatraman wrote: > Hi all, > > Currently, the only way to run distribute

Re: Understand Samza default metrics

2016-02-24 Thread Xinyu Liu
ss null > messages. So I don't trust the accuracy of that metric. > > Each metric emitted by Samza contains a header which includes job name, > job > id, container name and metric timestamp. You can use it to calculate > messages per second values. > > If you are using K

Re: Understand Samza default metrics

2016-02-23 Thread xinyu liu
Hi, David, I didn't find a wiki page that contains the descriptions of all Samza metrics. You can find the basic metrics by googling the following classes: SamzaContainerMetrics, TaskInstanceMetrics, SystemConsumersMetrics and SystemProducersMetrics. For your example, you can use the "process-call

Re: Review Request 43350: SAMZA-867 Fix job restart/shutdown in the event of a node outage.

2016-02-23 Thread Xinyu Liu
ception.java (line 27) <https://reviews.apache.org/r/43350/#comment181838> is serialVersionUID needed here? If so please use the autogenerated one. - Xinyu Liu On Feb. 23, 2016, 9:43 p.m., Jake Maes wrote: > > --- &g

Re: Review Request 43053: allow warning instead of fail in case of invalid num of partitions in the checkpoint partition

2016-02-01 Thread Xinyu Liu
-kafka/src/main/scala/org/apache/samza/util/KafkaUtil.scala (line 162) <https://reviews.apache.org/r/43053/#comment178367> nitpick: the var name of "msg1" seems not very well chosen. Shall we do warn(msg + "...") instead? - Xinyu Liu On Feb. 1, 2016, 6:25

Review Request 41912: SAMZA-850: Yarn Job Validation Tool

2016-01-04 Thread Xinyu Liu
org/r/41912/diff/ Testing --- Unit tests added to cover all the new methods/classes. Thanks, Xinyu Liu

Re: [VOTE] Samza 0.10.0 Release Candidate 2

2015-12-09 Thread xinyu liu
+1 on my side. I also ran the gradle build and unit tests without failure. Thanks, Xinyu On Wed, Dec 9, 2015 at 9:54 AM, Tao Feng wrote: > +1 from my side(non-binding). I download the package and successfully run > all the unit tests without failure. > > On Tue, Dec 8, 2015 at 3:38 PM, Yi Pan

Review Request 41071: SAMZA-843: fix heap usage increase caused by container timer change

2015-12-07 Thread Xinyu Liu
ing --- Thanks, Xinyu Liu

Re: Review Request 39806: SAMZA-798 : Performance and stability issue after combining checkpoint and coordinator stream

2015-11-02 Thread Xinyu Liu
> On Oct. 30, 2015, 6:06 p.m., Xinyu Liu wrote: > > samza-core/src/main/scala/org/apache/samza/checkpoint/CheckpointTool.scala, > > line 132 > > <https://reviews.apache.org/r/39806/diff/1/?file=1112956#file1112956line132> > > > > I think scala pre

Re: Review Request 39806: SAMZA-798 : Performance and stability issue after combining checkpoint and coordinator stream

2015-10-30 Thread Xinyu Liu
123) <https://reviews.apache.org/r/39806/#comment162817> I think scala prefers to use the companion object as the factory to create new instance (code before change). Is there any reason for this change? - Xinyu Liu On Oct. 30, 2015, 7:09 a.m., N

<    1   2   3