Re: [VOTE] Apache Samza 1.7.0 RC1

2022-03-15 Thread Yi Pan
+1 (binding). Ran check-all, verified the signature and checksums. All passed. Thanks for pushing 1.7.0 out of the door! Yi On Fri, Mar 11, 2022 at 2:31 PM Xinyu Liu wrote: > +1 (binding). > > Verified the signature and checksums, and also ran check-all tests which > all passed. > > Thanks,

Re: [RESULT][VOTE] Apache Samza 1.7.0 RC1

2022-03-15 Thread Yi Pan
Thanks, Daniel! Just want to mention that Boris also voted +1 (binding). Best! -Yi On Tue, Mar 15, 2022 at 9:22 AM Daniel Chen wrote: > Hey all, > > The vote for 1.7.0 release has been out for more than 72 hours and we got > +1(binding) x3 from Yi, Xinyu, Daniel > > Samza 1.7.0 officially

[REPORT] Samza - Feb 2022

2022-02-09 Thread Yi Pan
## Description: The mission of Samza is the creation and maintenance of software related to distributed stream processing framework ## Issues: - There are no issues requiring board attention. ## Membership Data: Apache Samza was founded 2015-01-22 (7 years ago) There are currently 29 committers

Re: [VOTE] SEP-30: Support Updates in Table API

2022-01-24 Thread Yi Pan
Discussed and resolved the minor concerns offline. +1 (binding) for this one. Thanks! -Yi On Tue, Dec 21, 2021 at 1:28 PM Xinyu Liu wrote: > +1 on my side. > > Glad to see this feature coming. Please make sure the api changes are > reflected in the documents, e.g. >

Re: [DISCUSS] Apache Samza 1.7.0 RC0

2022-01-26 Thread Yi Pan
Huge +1! Can't wait to see this list of features coming out! -Yi On Wed, Jan 26, 2022 at 2:36 PM Daniel Chen wrote: > Hi folks, > > We have added a number of major features and changes to master since > > 1.6, that warrants a major 1.7 release. > > Within LinkedIn, some of these features have

[REPORT] Samza - April 2022

2022-04-13 Thread Yi Pan
## Description: The mission of Samza is the creation and maintenance of software related to distributed stream processing framework ## Issues: - There are no issues requiring board attention. ## Membership Data: Apache Samza was founded 2015-01-22 (7 years ago) There are currently 29 committers

Re: Running v1.7.0 locally

2022-09-02 Thread Yi Pan
Hey, Malcolm, Thanks for reporting this issue. Could you open a JIRA to track that? Best! -Yi On Mon, Aug 29, 2022 at 5:53 PM Malcolm McFarland wrote: > Hey folks, > > I've recently been attempting to upgrade our legacy application from Samza > 1.5.1 to 1.7.0. With version 1.5.1, I've had no

Re: Java 11 Checkin again

2022-09-02 Thread Yi Pan
Hey, James, Thanks for the ping. @prateek, can we have someone to review this change? One question: have you tested the change w/ the older YARN cluster version (running 2.10.1)? If this change requires YARN cluster upgrade to 3.3.4 as well, that may be a breaking change to existing Samza users

Re: Java 11 Checkin again

2022-09-19 Thread Yi Pan
, but just wanted to call it out. > > -Jamie > > > > On Wed, Sep 14, 2022 at 5:52 PM Yi Pan wrote: > > > Hey, James, > > > > In order to merge your PR without breaking the jdk8 older modules, we > will > > need the changes proposed here. Can

Re: Java 11 Checkin again

2022-09-20 Thread Yi Pan
; > > > > ./gradlew build -PscalaSuffix=2.11 > > > > > > The build fails with this command using that version of Java 8 ^. > > > > > > Anyway, just wanted to point that out since I hit this in my branch > > > trying to utilize the "bin/c

Re: Java 11 Checkin again

2022-09-14 Thread Yi Pan
it > >> would > >> > work. Yarn itself is not compatible at 2.10.1 with Java 11 so you > would > >> > have to update yarn even if the Java library here wasn't updated. > >> > > >> > Could we move this version I'm proposing to a 2.x vers

Re: Java 11 Checkin again

2022-09-14 Thread Yi Pan
Java 11 and yarn 3 module need to only use the Scala 2.12 > > version of the build. > > > > Jamie > > > > > > On Wed, Sep 14, 2022, 2:38 AM Yi Pan wrote: > > > >> Hi, James, > >> > >> Sorry to reply late. I just came back

Re: Request for new release

2022-10-15 Thread Yi Pan
Hi, James, Thanks for the reminder. We are preparing the new 1.8 release. It is expected to be the end of this quarter. Best, -Yi On Mon, Oct 10, 2022 at 12:52 PM James DeMichele wrote: > Hello, we just had a pr merged to main in the Samza app that now supports > Java 11 runtime

Re: [VOTE] Apache Samza 1.8.0 RC0

2023-01-06 Thread Yi Pan
(+1) binding, Downloaded the src tarball, run check-all.sh and passed all tests. One thing noticed: there are configurations for your personal keys used for publishing the jars checked in gradle.properties. I don't think that we need to include that in the published src tarball. Otherwise,

Re: [ANNOUNCE] Welcome Ajo Thomas as Samza Committer

2022-12-16 Thread Yi Pan
Welcome and congrats, Ajo! - Yi On Wed, Dec 14, 2022 at 3:42 PM Xinyu Liu wrote: > Hi, All, > > I am glad to announce that Ajo Thomas has officially accepted our > invitation and become an Apache Samza Committer now. > > Ajo has made contributions to improve both Samza user experience and >

Re: SEP-31: Pipeline Drain: Support the ability to drain pipelines to allow incompatible intermediate schema changes

2022-12-08 Thread Yi Pan
ready to shutdown, RunLoop > shuts down. > > Do you see any issues with it ? > > - Ajo > > > On Thu, 1 Dec 2022 at 20:06, Yi Pan wrote: > > > Hi, Ajo, > > > > Sorry to reply this late. Could you clarify one thing in the design: For > > watermark triggered

Re: [VOTE] SEP-31: Pipeline Drain- Support the ability to drain pipelines to allow incompatible intermediate schema changes

2022-12-08 Thread Yi Pan
+1. Long awaited feature! Thanks! -Yi On Tue, Nov 29, 2022 at 11:46 AM Xinyu Liu wrote: > +1. > > Overall the design looks good. Thanks for contributing to this feature. > > Thanks, > Xinyu > > On Tue, Nov 29, 2022 at 10:44 AM Ajo Thomas > wrote: > > > Hi All, > > > > This is a call for a

Re: SEP-31: Pipeline Drain: Support the ability to drain pipelines to allow incompatible intermediate schema changes

2022-12-01 Thread Yi Pan
Hi, Ajo, Sorry to reply this late. Could you clarify one thing in the design: For watermark triggered window draining, is the infinitive watermark trigger happen first, or the drain token in all source SSP happen first? Shouldn't it be the following sequence: a) all drain token from all input

Re: [DISCUSS] SEP-32: Elasticity for Samza

2023-01-19 Thread Yi Pan
Hey, Manasa, Sorry to chime in late. A few questions: a) how are states for the virtual tasks managed during split/merge? b) what's perf impact when we have 2 virtual tasks on the same SSP in the same container, while one virtual task is much faster than the other? c) what's the reason that a

[REPORT] Samza - Nov 2022

2022-11-09 Thread Yi Pan
## Description: The mission of Samza is the creation and maintenance of software related to distributed stream processing framework ## Issues: - There are no issues requiring board attention. ## Membership Data: Apache Samza was founded 2015-01-22 (8 years ago) There are currently 29 committers

Re: [DISCUSS] SEP-32: Elasticity for Samza

2023-02-06 Thread Yi Pan
(single task) in key ordered executor >> sitting in the same host will increase the load on the host and (c) if one >> or more of the parallel units (threads here) needs more resources, it will >> result in large container which makes scheduling harder as finding large >&

Re: [VOTE] SEP-32: Elasticity for Samza

2023-02-08 Thread Yi Pan
+1 (binding) Thanks! -Yi On Tue, Feb 7, 2023 at 2:14 PM Bharath Kumara Subramanian < codin.mart...@gmail.com> wrote: > +1 (binding) > > Cheers, > Bharath > > On Tue, Feb 7, 2023 at 12:56 PM Lakshmi Manasa > wrote: > > > Hi folks, > > > > This is a call for vote on SEP-32: Elasticity for

Re: Review Request 30287: Stream SQL Object Model Draft

2015-01-27 Thread Yi Pan (Data Infrastructure)
and join expression are expressions that all generate a table-type data source entity. - Yi Pan (Data Infrastructure) On Jan. 26, 2015, 10:04 p.m., Milinda Pathirage wrote: --- This is an automatically generated e-mail. To reply, visit

Re: Review Request 29754: StreamSQL operator API draft

2015-01-30 Thread Yi Pan (Data Infrastructure)
/task/sql/StreamSqlTask.java PRE-CREATION settings.gradle 3a01fd66359b8c79954ae8f34eeaf4b2e3fdc0b4 Diff: https://reviews.apache.org/r/29754/diff/ Testing --- run ./bin/check-all.sh passed Thanks, Yi Pan (Data Infrastructure)

Re: Review Request 31909: SAMZA-590

2015-03-11 Thread Yi Pan (Data Infrastructure)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31909/#review76120 --- Ship it! Ship It! - Yi Pan (Data Infrastructure) On March 10

Re: Review Request 32006: SAMZA-597

2015-03-13 Thread Yi Pan (Data Infrastructure)
/apache/samza/logging/log4j/serializers/LoggingEventJsonSerde.java https://reviews.apache.org/r/32006/#comment123934 nit: SerdeObject - Yi Pan (Data Infrastructure) On March 13, 2015, 12:57 a.m., Chris Riccomini wrote

Re: Review Request 32052: SAMZA-592

2015-03-13 Thread Yi Pan (Data Infrastructure)
in the cache, getOffsets would raise exception? And how do we capture that case? - Yi Pan (Data Infrastructure) On March 13, 2015, 7:56 p.m., Chris Riccomini wrote: --- This is an automatically generated e-mail. To reply, visit: https

Re: Review Request 32006: SAMZA-597

2015-03-13 Thread Yi Pan (Data Infrastructure)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32006/#review76430 --- Ship it! Ship It! - Yi Pan (Data Infrastructure) On March 13

Re: Review Request 32052: SAMZA-592

2015-03-13 Thread Yi Pan (Data Infrastructure)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32052/#review76442 --- Ship it! Ship It! - Yi Pan (Data Infrastructure) On March 13

Re: Review Request 32052: SAMZA-592

2015-03-13 Thread Yi Pan (Data Infrastructure)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32052/#review76437 --- Ship it! Ship It! - Yi Pan (Data Infrastructure) On March 13

Re: Review Request 32407: SAMZA-571: add suppression interface for uncaught exceptions

2015-03-24 Thread Yi Pan (Data Infrastructure)
samza-core/src/test/scala/org/apache/samza/container/TestTaskInstance.scala 54b4df84f47f818d62ac0361196567ad1f430fde Diff: https://reviews.apache.org/r/32407/diff/ Testing (updated) --- Unit tests added. Pass with ./bin/check-all.sh Thanks, Yi Pan (Data Infrastructure)

Review Request 33142: [SAMZA-561] Review in progress

2015-04-13 Thread Yi Pan (Data Infrastructure)
samza-test/src/main/python/tests/sql_tests.py PRE-CREATION samza-test/src/main/resources/orders.avsc PRE-CREATION samza-test/src/main/resources/orders.json PRE-CREATION Diff: https://reviews.apache.org/r/33142/diff/ Testing --- Thanks, Yi Pan (Data Infrastructure)

Re: Review Request 33146: New KeyValueStore Features

2015-04-21 Thread Yi Pan (Data Infrastructure)
/TestKeyValueStores.scala https://reviews.apache.org/r/33146/#comment131257 nit: prefer not to re-order the methods if not necessary. - Yi Pan (Data Infrastructure) On April 16, 2015, 10:43 a.m., Mohamed Mahmoud (El-Geish) wrote

Re: Review Request 33146: New KeyValueStore Features

2015-04-26 Thread Yi Pan (Data Infrastructure)
On April 24, 2015, 5:01 p.m., Yi Pan (Data Infrastructure) wrote: Ship It! Mohamed Mahmoud (El-Geish) wrote: I don't have access to commit. Can you please grant me access or commit for me? Thanks! Hi, MOhamed, I was trying to go through all the tests w/ your patch. After all

Re: Review Request 33146: New KeyValueStore Features

2015-04-21 Thread Yi Pan (Data Infrastructure)
On April 21, 2015, 6:49 p.m., Yi Pan (Data Infrastructure) wrote: samza-kv/src/main/java/org/apache/samza/storage/kv/KeyValueStore.java, line 33 https://reviews.apache.org/r/33146/diff/2/?file=931566#file931566line33 The signature of close() and flush() functions from

Re: Review Request 33146: New KeyValueStore Features

2015-04-24 Thread Yi Pan (Data Infrastructure)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33146/#review81497 --- Ship it! Ship It! - Yi Pan (Data Infrastructure) On April 24

Re: Review Request 33761: Fix SAMZA-658

2015-05-04 Thread Yi Pan (Data Infrastructure)
/KeyValueStorageEngine.scala https://reviews.apache.org/r/33761/#comment133095 nit: same here. - Yi Pan (Data Infrastructure) On May 1, 2015, 6:43 p.m., Guozhang Wang wrote: --- This is an automatically generated e-mail. To reply, visit: https

Re: Review Request 33453: SAMZA-557 Reuse local state in SamzaContainer on clean shutdown

2015-04-28 Thread Yi Pan (Data Infrastructure)
. - Yi Pan (Data Infrastructure) On April 22, 2015, 9:54 p.m., Navina Ramesh wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33453

Re: Review Request 33749: WIP: SAMZA-650 window store implementation

2015-05-07 Thread Yi Pan (Data Infrastructure)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33749/#review82821 --- On May 4, 2015, 6:58 a.m., Yi Pan (Data Infrastructure) wrote

Re: Review Request 33749: WIP: SAMZA-650 window store implementation

2015-05-07 Thread Yi Pan (Data Infrastructure)
. To reply, visit: https://reviews.apache.org/r/33749/#review82824 --- On May 4, 2015, 6:58 a.m., Yi Pan (Data Infrastructure) wrote: --- This is an automatically generated e-mail

Re: Review Request 33488: SAMZA-657

2015-05-07 Thread Yi Pan (Data Infrastructure)
/samza/test/integration/join/Emitter.java https://reviews.apache.org/r/33488/#comment133628 nit: There are still many trailing white spaces. We should remove them. - Yi Pan (Data Infrastructure) On April 27, 2015, 7:59 p.m., Guozhang Wang wrote

Re: Review Request 33761: Fix SAMZA-658

2015-05-06 Thread Yi Pan (Data Infrastructure)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33761/#review82777 --- Ship it! Ship It! - Yi Pan (Data Infrastructure) On May 6, 2015

Re: Review Request 33453: SAMZA-557 Reuse local state in SamzaContainer on clean shutdown

2015-05-06 Thread Yi Pan (Data Infrastructure)
-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala https://reviews.apache.org/r/33453/#comment133550 Should be default here. - Yi Pan (Data Infrastructure) On May 6, 2015, 6:22 a.m., Navina Ramesh wrote

Re: Review Request 33749: WIP: SAMZA-650 window store implementation

2015-05-06 Thread Yi Pan (Data Infrastructure)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33749/#review82674 --- On May 4, 2015, 6:58 a.m., Yi Pan (Data Infrastructure) wrote

Re: Review Request 33749: WIP: SAMZA-650 window store implementation

2015-05-06 Thread Yi Pan (Data Infrastructure)
--- On May 4, 2015, 6:58 a.m., Yi Pan (Data Infrastructure) wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33749

Re: Review Request 33146: New KeyValueStore Features

2015-05-04 Thread Yi Pan (Data Infrastructure)
getAll() vs many get() that directly hitting RocksDB APIs? - Yi Pan (Data Infrastructure) On May 4, 2015, 4:27 a.m., Mohamed Mahmoud (El-Geish) wrote: --- This is an automatically generated e-mail. To reply, visit: https

Re: Review Request 33146: New KeyValueStore Features

2015-05-04 Thread Yi Pan (Data Infrastructure)
On May 4, 2015, 8:14 p.m., Yi Pan (Data Infrastructure) wrote: samza-test/src/main/scala/org/apache/samza/test/performance/TestKeyValuePerformance.scala, line 320 https://reviews.apache.org/r/33146/diff/5-6/?file=943969#file943969line320 This is a bit confusing to me: why do we

Review Request 34009: WIP: SAMZA-650 window store implementation

2015-05-08 Thread Yi Pan (Data Infrastructure)
/StreamSqlTask.java PRE-CREATION samza-sql-core/src/test/java/org/apache/samza/task/sql/UserCallbacksSqlTask.java PRE-CREATION Diff: https://reviews.apache.org/r/34009/diff/ Testing --- Thanks, Yi Pan (Data Infrastructure)

Re: Review Request 34009: WIP: SAMZA-650 window store implementation

2015-05-08 Thread Yi Pan (Data Infrastructure)
/test/java/org/apache/samza/task/sql/UserCallbacksSqlTask.java PRE-CREATION Diff: https://reviews.apache.org/r/34009/diff/ Testing (updated) --- ./gradlew clean build passed Thanks, Yi Pan (Data Infrastructure)

Re: Review Request 34009: WIP: SAMZA-650 window store implementation

2015-05-12 Thread Yi Pan (Data Infrastructure)
--- On May 9, 2015, 1:52 a.m., Yi Pan (Data Infrastructure) wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34009

Re: Review Request 34009: WIP: SAMZA-650 window store implementation

2015-05-12 Thread Yi Pan (Data Infrastructure)
://reviews.apache.org/r/34009/#review83395 --- On May 9, 2015, 1:52 a.m., Yi Pan (Data Infrastructure) wrote: --- This is an automatically generated e-mail. To reply, visit: https

Re: Review Request 34009: WIP: SAMZA-650 window store implementation

2015-05-12 Thread Yi Pan (Data Infrastructure)
, 2015, 1:52 a.m., Yi Pan (Data Infrastructure) wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34009/ --- (Updated May 9, 2015, 1

Re: Review Request 33735: RocksDB TTL support

2015-05-14 Thread Yi Pan (Data Infrastructure)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33735/#review83847 --- Ship it! Ship It! - Yi Pan (Data Infrastructure) On May 13

Re: Review Request 34206: WIP: update operator API to allow callbacks and allow a single API to trigger OperatorRouter execution w/ user callbacks

2015-05-14 Thread Yi Pan (Data Infrastructure)
/UserCallbacksSqlTask.java PRE-CREATION Diff: https://reviews.apache.org/r/34206/diff/ Testing --- Thanks, Yi Pan (Data Infrastructure)

Re: Review Request 34206: WIP: update operator API to allow callbacks and allow a single API to trigger OperatorRouter execution w/ user callbacks

2015-05-14 Thread Yi Pan (Data Infrastructure)
--- Thanks, Yi Pan (Data Infrastructure)

Re: Review Request 34009: SAMZA-552 window store implementation

2015-05-13 Thread Yi Pan (Data Infrastructure)
. - Yi --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34009/#review83667 --- On May 13, 2015, 5:36 p.m., Yi Pan (Data Infrastructure) wrote

Review Request 34206: WIP: update operator API to allow callbacks and allow a single API to trigger OperatorRouter execution w/ user callbacks

2015-05-14 Thread Yi Pan (Data Infrastructure)
/test/java/org/apache/samza/task/sql/UserCallbacksSqlTask.java PRE-CREATION Diff: https://reviews.apache.org/r/34206/diff/ Testing --- Thanks, Yi Pan (Data Infrastructure)

Re: Review Request 34206: WIP: update operator API to allow callbacks and allow a single API to trigger OperatorRouter execution w/ user callbacks

2015-05-15 Thread Yi Pan (Data Infrastructure)
/#review83937 --- On May 15, 2015, 2:16 a.m., Yi Pan (Data Infrastructure) wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34206

Re: Review Request 33170: Renamed samza-sql to samza-sql-core

2015-04-14 Thread Yi Pan (Data Infrastructure)
-calcite samza-sql-core/src/test/java/org/apache/samza/sql/test/metadata/TestAvroSchemaConverter.java https://reviews.apache.org/r/33170/#comment129816 Same here. - Yi Pan (Data Infrastructure) On April 14, 2015, 3:14 p.m., Milinda Pathirage wrote

Re: Review Request 33219: [SAMZA-649] Create samza-sql-calcite module for Calcite SQL front end

2015-04-15 Thread Yi Pan (Data Infrastructure)
On April 15, 2015, 6:20 p.m., Yi Pan (Data Infrastructure) wrote: samza-sql-calcite/src/main/java/org/apache/samza/sql/calcite/schema/AvroSchemaConverter.java, line 37 https://reviews.apache.org/r/33219/diff/1/?file=930371#file930371line37 I assume that this class is used

Re: Review Request 33219: [SAMZA-649] Create samza-sql-calcite module for Calcite SQL front end

2015-04-15 Thread Yi Pan (Data Infrastructure)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33219/#review80237 --- Ship it! +1 - Yi Pan (Data Infrastructure) On April 15, 2015, 2

Re: Review Request 33142: [SAMZA-561] Review in progress

2015-04-14 Thread Yi Pan (Data Infrastructure)
. samza-sql/src/main/java/org/apache/samza/sql/operators/scan/ProjectableFilterableStreamScanSpec.java https://reviews.apache.org/r/33142/#comment129692 These calcite specific class should be moved out-of samza-sql-core module. - Yi Pan (Data Infrastructure) On April 13, 2015, 9:04 p.m., Yi

Review Request 32872: SAMZA-571: add test to capture RecordTooLargeException

2015-04-06 Thread Yi Pan (Data Infrastructure)
d66b3bd070a4cef4b1d3dded1d79a33cbe3fa09b Diff: https://reviews.apache.org/r/32872/diff/ Testing --- Passed local test suite Thanks, Yi Pan (Data Infrastructure)

Re: Review Request 35723: SAMZA-720: fix bootstrap hangs when container number 1

2015-06-22 Thread Yi Pan (Data Infrastructure)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35723/#review88820 --- Ship it! +1. LGTM. Thanks for the quick fix, Yan. - Yi Pan (Data

Re: Review Request 35397: Fix SAMZA-697

2015-06-19 Thread Yi Pan (Data Infrastructure)
{ task.process(envelope, collector, coordinator) } } ... - Yi Pan (Data Infrastructure) On June 18, 2015, 6:42 p.m., Guozhang Wang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r

Re: Review Request 35325: SAMZA-698: update Samza and Spark Streaming Comparison

2015-06-12 Thread Yi Pan (Data Infrastructure)
to claim exact-once under the asumption that the system running healthy, - Yi Pan (Data Infrastructure) On June 12, 2015, 11:54 p.m., Yan Fang wrote: --- This is an automatically generated e-mail. To reply, visit: https

Re: Review Request 34500: SAMZA-552 Operator API change: builder and simplified operator classes

2015-05-29 Thread Yi Pan (Data Infrastructure)
completely remove that. Yi Pan (Data Infrastructure) wrote: Thank you both for the good points here. @Navina, yes, the basic idea for the topology builder is exactly what you mentioned and the model you illustrated is much simpler and very attractive. The issue I saw

Re: Review Request 34746: Adding new CoordinatorStreamMessage SetContainerHostMapping and LocalityManager (SAMZA-618)

2015-06-01 Thread Yi Pan (Data Infrastructure)
On May 30, 2015, 8:58 a.m., Yi Pan (Data Infrastructure) wrote: samza-core/src/main/java/org/apache/samza/container/LocalityManager.java, line 62 https://reviews.apache.org/r/34746/diff/2/?file=974783#file974783line62 This would be invoked twice by checkpointManager

Re: Review Request 34746: Adding new CoordinatorStreamMessage SetContainerHostMapping and LocalityManager (SAMZA-618)

2015-06-01 Thread Yi Pan (Data Infrastructure)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34746/#review86125 --- Ship it! Ship It! - Yi Pan (Data Infrastructure) On May 30

Review Request 34664: SAMZA-552 Operator API change: builder and simplified operator classes

2015-05-26 Thread Yi Pan (Data Infrastructure)
commit of the following: commit fad81106901e494d3950eeaafaeefef482ac0125 Author: Yi Pan (Data Infrastructure) yi...@linkedin.com Date: Mon May 25 23:40:00 2015 -0700 SAMZA-650 window message store and window store implementation commit 58c2eeebf4bb0975f70aeba733379e1104f3a7de Author: Yi Pan

Review Request 34574: SAMZA-608; don't hange on serde errors in system consumers

2015-05-21 Thread Yi Pan (Data Infrastructure)
/SystemConsumers.scala 125d37602e2c0a9da75674f37580a1ac02f94796 samza-core/src/test/scala/org/apache/samza/system/TestSystemConsumers.scala 3fdc781c1275f928f4b51b01869e1122502a2c08 Diff: https://reviews.apache.org/r/34574/diff/ Testing --- Thanks, Yi Pan (Data Infrastructure)

Re: Review Request 36006: Writing a tool to read from the coordinator stream and react to config changes accordingly.

2015-07-31 Thread Yi Pan (Data Infrastructure)
/SamzaAppMasterService.scala (line 61) https://reviews.apache.org/r/36006/#comment148217 Can we use CoordinatorStreamMessage defined constants, instead of hard-code strings here? - Yi Pan (Data Infrastructure) On July 28, 2015, 2:39 a.m., Shadi A. Noghabi wrote

Re: Review Request 37069: SAMZA-738 Samza Timer based metrics does not have enough precision

2015-08-18 Thread Yi Pan (Data Infrastructure)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37069/#review95692 --- Ship it! LGTM. - Yi Pan (Data Infrastructure) On Aug. 5, 2015

Re: Review Request 37506: WIP: SAMZA-552 Operator API change: New Builder API

2015-08-24 Thread Yi Pan (Data Infrastructure)
of requiring re-implementing the TopologyBuilder layer later. - Yi Pan (Data Infrastructure) On Aug. 16, 2015, 3:57 p.m., Milinda Pathirage wrote: --- This is an automatically generated e-mail. To reply, visit: https

Re: Review Request 37528: SAMZA-736 BrokerProxy will stuck in infinite loop if consumer.fetch throws OOME

2015-08-20 Thread Yi Pan (Data Infrastructure)
/samza/system/kafka/TestBrokerProxy.scala (line 307) https://reviews.apache.org/r/37528/#comment151197 nit: trailing white space. - Yi Pan (Data Infrastructure) On Aug. 19, 2015, 9:03 a.m., Aleksandar Pejakovic wrote

Re: Review Request 36903: SAMZA-744: shutdown stores before shutdown producers

2015-07-29 Thread Yi Pan (Data Infrastructure)
producers Diffs - samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 27b2517048ad5730762506426ee7578c66181db8 Diff: https://reviews.apache.org/r/36903/diff/ Testing (updated) --- ./bin/check-all.sh passed Thanks, Yi Pan (Data Infrastructure)

Re: Review Request 36905: SAMZA-745 elasticsearch module has Javadoc warning

2015-07-29 Thread Yi Pan (Data Infrastructure)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36905/#review93479 --- Ship it! Ship It! - Yi Pan (Data Infrastructure) On July 29

Review Request 36903: SAMZA-744: WIP patch

2015-07-28 Thread Yi Pan (Data Infrastructure)
27b2517048ad5730762506426ee7578c66181db8 Diff: https://reviews.apache.org/r/36903/diff/ Testing --- Thanks, Yi Pan (Data Infrastructure)

Re: Review Request 36006: Writing a tool to read from the coordinator stream and react to config changes accordingly.

2015-07-29 Thread Yi Pan (Data Infrastructure)
On July 22, 2015, 7:08 p.m., Yi Pan (Data Infrastructure) wrote: samza-core/src/main/java/org/apache/samza/autoScaling/deployer/ConfigManager.java, line 318 https://reviews.apache.org/r/36006/diff/2/?file=1015151#file1015151line318 Did you update the container count

Re: Review Request 36815: SAMZA-741 Support for versioning with Elasticsearch Producer

2015-07-29 Thread Yi Pan (Data Infrastructure)
On July 29, 2015, 5:47 a.m., Yi Pan (Data Infrastructure) wrote: samza-elasticsearch/src/main/java/org/apache/samza/system/elasticsearch/ElasticsearchSystemProducer.java, line 149 https://reviews.apache.org/r/36815/diff/4/?file=1024086#file1024086line149 Quick question

Re: Review Request 36163: SAMZA-690: changelog topic creation should not be in the container code

2015-08-05 Thread Yi Pan (Data Infrastructure)
On July 27, 2015, 7:36 a.m., Yi Pan (Data Infrastructure) wrote: The code LGTM. For testing, if we can verify this fix w/ a stateful StreamTask w/ changelog enabled with some partition numbers that are different from the default auto-creation partition number (i.e. 8) in Kafka

Re: Review Request 37102: SAMZA-753: BrokerProxy stop should shutdown kafka consumer first

2015-08-04 Thread Yi Pan (Data Infrastructure)
test? It would be good to verify it. - Yi Pan (Data Infrastructure) On Aug. 4, 2015, 11:07 p.m., Yan Fang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37102

Re: Review Request 37069: SAMZA-738 Samza Timer based metrics does not have enough precision

2015-08-04 Thread Yi Pan (Data Infrastructure)
. updateTimerAndGetDuration internally calls clock(), and we are calling clock() twice here as well. Should at least cache the return value from clock() and re-use it here. - Yi Pan (Data Infrastructure) On Aug. 4, 2015, 9:18 a.m., Aleksandar Pejakovic wrote

Re: Review Request 36903: SAMZA-744: shutdown stores before shutdown producers

2015-08-04 Thread Yi Pan (Data Infrastructure)
/src/test/scala/org/apache/samza/test/integration/TestTask.scala PRE-CREATION Diff: https://reviews.apache.org/r/36903/diff/ Testing --- ./bin/check-all.sh passed Thanks, Yi Pan (Data Infrastructure)

Re: Review Request 36903: SAMZA-744: shutdown stores before shutdown producers

2015-08-06 Thread Yi Pan (Data Infrastructure)
-all.sh passed Thanks, Yi Pan (Data Infrastructure)

Re: Review Request 36903: SAMZA-744: shutdown stores before shutdown producers

2015-08-06 Thread Yi Pan (Data Infrastructure)
; Fixed - Yi --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36903/#review94193 --- On Aug. 6, 2015, 11:24 a.m., Yi Pan (Data Infrastructure

Re: Review Request 36006: Writing a tool to read from the coordinator stream and react to config changes accordingly.

2015-07-22 Thread Yi Pan (Data Infrastructure)
us to keep the minimum dependencies in samza-core. I will continue w/ the implementation of ConfigManager. - Yi Pan (Data Infrastructure) On July 18, 2015, 2:52 a.m., Shadi A. Noghabi wrote: --- This is an automatically generated e-mail

Re: Review Request 36006: Writing a tool to read from the coordinator stream and react to config changes accordingly.

2015-07-22 Thread Yi Pan (Data Infrastructure)
How is it used? I don't see it used in ConfigManager. Where else could it be used? If no one uses it, we should remove this function. - Yi Pan (Data Infrastructure) On July 18, 2015, 2:52 a.m., Shadi A. Noghabi wrote

Re: Review Request 36163: SAMZA-690: changelog topic creation should not be in the container code

2015-07-27 Thread Yi Pan (Data Infrastructure)
/KafkaSystemAdmin.scala (line 400) https://reviews.apache.org/r/36163/#comment147350 Nit: It would be good to put a note here: validateChangelogStream() should not be called before createChangelogStream(), before Kafka fixes the admin API to add a read-only flag in fetchMetadata(). - Yi Pan (Data

Re: Review Request 39558: Fix parsing errors in broadcast stream config values

2015-10-22 Thread Yi Pan (Data Infrastructure)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/39558/#review103657 --- Ship it! Ship It! - Yi Pan (Data Infrastructure) On Oct. 22

Re: Review Request 39252: SAMZA-626 - tool to read the RocksDb in a running job (Yan's patch)

2015-10-22 Thread Yi Pan (Data Infrastructure)
t we validate only one keyArgu is included in the options? samza-kv-rocksdb/src/main/java/org/apache/samza/storage/kv/RocksDbReadingTool.java (line 96) <https://reviews.apache.org/r/39252/#comment161776> nit: Since the db-path and db-name are required options, the if conditions seem

Review Request 39575: SAMZA-793 static config rewriter should be invoked in JobRunner

2015-10-22 Thread Yi Pan (Data Infrastructure)
a926ce639dbf6776ebfb1d885621f4b6bf5f5aa5 samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala d6109ec4b3eedfe32325c6dc420c6c8bb18eb79b Diff: https://reviews.apache.org/r/39575/diff/ Testing --- Verified via test jobs. Thanks, Yi Pan (Data Infrastructure)

Re: Review Request 39464: SAMZA-723 - hello-samza hangs when we use StreamAppender (Yan's patch)

2015-10-23 Thread Yi Pan (Data Infrastructure)
e code here to grab the configuration? Then, we don't even need this if-else condition here, since the caller should be aware of the process context this getConfig is called and supply the cooresponding config differently. However, this could be a later code cleanup task. - Yi Pan (Data Infrastr

Re: Review Request 39464: SAMZA-723 - hello-samza hangs when we use StreamAppender (Yan's patch)

2015-10-23 Thread Yi Pan (Data Infrastructure)
method. - Yi Pan (Data Infrastructure) On Oct. 23, 2015, 6:33 p.m., Navina Ramesh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.

Re: Review Request 39252: SAMZA-626 - tool to read the RocksDb in a running job (Yan's patch)

2015-10-26 Thread Yi Pan (Data Infrastructure)
tional. - Yi Pan (Data Infrastructure) On Oct. 23, 2015, 11:46 p.m., Navina Ramesh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.

Re: Review Request 40106: SAMZA-812 CachedStore flushes too often

2015-11-11 Thread Yi Pan (Data Infrastructure)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/40106/#review106084 --- Ship it! Ship It! - Yi Pan (Data Infrastructure) On Nov. 9

Re: Review Request 40106: SAMZA-812 CachedStore flushes too often

2015-11-11 Thread Yi Pan (Data Infrastructure)
> On Nov. 11, 2015, 7:31 a.m., Yi Pan (Data Infrastructure) wrote: > > samza-kv/src/test/scala/org/apache/samza/storage/kv/TestCachedStore.scala, > > line 37 > > <https://reviews.apache.org/r/40106/diff/1/?file=1120744#file1120744line37> > > > > q

Re: Review Request 40034: added list-yarn-job.sh

2015-11-09 Thread Yi Pan (Data Infrastructure)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/40034/#review105721 --- Ship it! Ship It! - Yi Pan (Data Infrastructure) On Nov. 6

Re: Review Request 39119: SAMZA-792: SamzaAppMaster Java code needs to pass the requested container memory size to RM

2015-10-08 Thread Yi Pan (Data Infrastructure)
nstead of the default ones. - Yi Pan (Data Infrastructure) On Oct. 8, 2015, 7:39 a.m., Navina Ramesh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.

Re: Review Request 39119: SAMZA-792: SamzaAppMaster Java code needs to pass the requested container memory size to RM

2015-10-08 Thread Yi Pan (Data Infrastructure)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/39119/#review101974 --- Ship it! Ship It! - Yi Pan (Data Infrastructure) On Oct. 8

<    1   2   3   4   5   6   7   8   >