Re: Review Request 31836: [HELIX-573] Add support to automatically compress/uncompress data in Zookeeper
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31836/ --- (Updated March 11, 2015, 3:49 p.m.) Review request for helix. Changes --- incorporated the feedback, also added the code in external view compute stage to copy the enableCompression from Idealstate to ExternalView Repository: helix-git Description --- commit 1ef44c2e9a132df3513a51e3a8dac658236a2263 Author: Kishore Gopalakrishna g.kish...@gmail.com Date: Sun Mar 8 16:40:29 2015 -0700 [HELIX-573] Add support to automatically compress/uncompress data in Zookeeper :100644 100644 4419fdd... 1f34529... M helix-core/src/main/java/org/apache/helix/manager/zk/ZNRecordSerializer.java :100644 100644 2d7cb3c... 26d7e2b... M helix-core/src/main/java/org/apache/helix/manager/zk/ZNRecordStreamingSerializer.java :00 100644 000... 90c1e8e... A helix-core/src/test/java/org/apache/helix/manager/zk/TestZNRecordSerializer.java :100644 100644 e4b0b25... 95064f8... M helix-core/src/test/java/org/apache/helix/manager/zk/TestZNRecordStreamingSerializer.java :100755 100755 e869f25... 99ef81c... M hpost-review.sh Diffs (updated) - helix-core/src/main/java/org/apache/helix/controller/stages/ExternalViewComputeStage.java 903840c helix-core/src/main/java/org/apache/helix/manager/zk/ZNRecordSerializer.java 4419fdd helix-core/src/main/java/org/apache/helix/manager/zk/ZNRecordStreamingSerializer.java 2d7cb3c helix-core/src/main/java/org/apache/helix/util/GZipCompressionUtil.java PRE-CREATION helix-core/src/test/java/org/apache/helix/integration/TestEnableCompression.java PRE-CREATION helix-core/src/test/java/org/apache/helix/manager/zk/TestZNRecordSerializer.java PRE-CREATION helix-core/src/test/java/org/apache/helix/manager/zk/TestZNRecordStreamingSerializer.java e4b0b25 Diff: https://reviews.apache.org/r/31836/diff/ Testing --- Added basic test for compress/uncompress Thanks, Kishore Gopalakrishna
Re: Review Request 31836: [HELIX-573] Add support to automatically compress/uncompress data in Zookeeper
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31836/#review76077 --- Ship it! helix-core/src/main/java/org/apache/helix/controller/stages/ExternalViewComputeStage.java https://reviews.apache.org/r/31836/#comment123472 whitespace helix-core/src/main/java/org/apache/helix/manager/zk/ZNRecordStreamingSerializer.java https://reviews.apache.org/r/31836/#comment123474 indentation helix-core/src/main/java/org/apache/helix/util/GZipCompressionUtil.java https://reviews.apache.org/r/31836/#comment123473 License should come before imports. helix-core/src/main/java/org/apache/helix/util/GZipCompressionUtil.java https://reviews.apache.org/r/31836/#comment123475 whitespace helix-core/src/test/java/org/apache/helix/integration/TestEnableCompression.java https://reviews.apache.org/r/31836/#comment123476 whitespace helix-core/src/test/java/org/apache/helix/integration/TestEnableCompression.java https://reviews.apache.org/r/31836/#comment123477 whitespace - Kanak Biscuitwala On March 11, 2015, 8:49 a.m., Kishore Gopalakrishna wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31836/ --- (Updated March 11, 2015, 8:49 a.m.) Review request for helix. Repository: helix-git Description --- commit 1ef44c2e9a132df3513a51e3a8dac658236a2263 Author: Kishore Gopalakrishna g.kish...@gmail.com Date: Sun Mar 8 16:40:29 2015 -0700 [HELIX-573] Add support to automatically compress/uncompress data in Zookeeper :100644 100644 4419fdd... 1f34529... M helix-core/src/main/java/org/apache/helix/manager/zk/ZNRecordSerializer.java :100644 100644 2d7cb3c... 26d7e2b... M helix-core/src/main/java/org/apache/helix/manager/zk/ZNRecordStreamingSerializer.java :00 100644 000... 90c1e8e... A helix-core/src/test/java/org/apache/helix/manager/zk/TestZNRecordSerializer.java :100644 100644 e4b0b25... 95064f8... M helix-core/src/test/java/org/apache/helix/manager/zk/TestZNRecordStreamingSerializer.java :100755 100755 e869f25... 99ef81c... M hpost-review.sh Diffs - helix-core/src/main/java/org/apache/helix/controller/stages/ExternalViewComputeStage.java 903840c helix-core/src/main/java/org/apache/helix/manager/zk/ZNRecordSerializer.java 4419fdd helix-core/src/main/java/org/apache/helix/manager/zk/ZNRecordStreamingSerializer.java 2d7cb3c helix-core/src/main/java/org/apache/helix/util/GZipCompressionUtil.java PRE-CREATION helix-core/src/test/java/org/apache/helix/integration/TestEnableCompression.java PRE-CREATION helix-core/src/test/java/org/apache/helix/manager/zk/TestZNRecordSerializer.java PRE-CREATION helix-core/src/test/java/org/apache/helix/manager/zk/TestZNRecordStreamingSerializer.java e4b0b25 Diff: https://reviews.apache.org/r/31836/diff/ Testing --- Added basic test for compress/uncompress Thanks, Kishore Gopalakrishna
RE: Cutting a release 0.6.5 tonight
Hi, Kishore I saw there are regression test failures from last two recent commits on 0.6.x branch, running from both local box and Linkedin's hudson jobs. Are we going to fix them before the release? Thanks Lei -- Lei Xia Software Engineer Data Infrastructure/Distributed Data Systems/Nuage LinkedIn l...@linkedin.com www.linkedin.com/in/lxia1 From: kishore g [g.kish...@gmail.com] Sent: Wednesday, March 11, 2015 12:04 PM To: u...@helix.apache.org Cc: dev@helix.apache.org Subject: Re: Cutting a release 0.6.5 tonight Will write a test case for migration and document the steps On Mar 11, 2015 11:37 AM, Varun Sharma va...@pinterest.com wrote: What would be the migration path from non-compressed buckets to compressed non bucket resources ? It seems even the CURRENTSTATES are being bucketed in this case, I thought that was not expected with bucketing. Does the controller read these current states appropriately ? To migrate, it seems that we would need to also rewrite the CURRENT STATES ? On Wed, Mar 11, 2015 at 10:02 AM, kishore g g.kish...@gmail.com wrote: Hi, I will work with Jason to cut a 0.6.5 release tonight. The new thing I added is to enableCompression while storing data in Zookeeper, this allows us to go up to 100k partitions per resource without having to use bucketing feature. We also fixed few bugs with bucketed resource just in case some one needs it. The property store api needs some changes, I plan to get it in today. Let me know if you need any other changes to be included. Are there any changes that went into 0.7.x branch that we need to merge it back in to 0.6.x ? thanks, Kishore G
Re: Cutting a release 0.6.5 tonight
Will write a test case for migration and document the steps On Mar 11, 2015 11:37 AM, Varun Sharma va...@pinterest.com wrote: What would be the migration path from non-compressed buckets to compressed non bucket resources ? It seems even the CURRENTSTATES are being bucketed in this case, I thought that was not expected with bucketing. Does the controller read these current states appropriately ? To migrate, it seems that we would need to also rewrite the CURRENT STATES ? On Wed, Mar 11, 2015 at 10:02 AM, kishore g g.kish...@gmail.com wrote: Hi, I will work with Jason to cut a 0.6.5 release tonight. The new thing I added is to enableCompression while storing data in Zookeeper, this allows us to go up to 100k partitions per resource without having to use bucketing feature. We also fixed few bugs with bucketed resource just in case some one needs it. The property store api needs some changes, I plan to get it in today. Let me know if you need any other changes to be included. Are there any changes that went into 0.7.x branch that we need to merge it back in to 0.6.x ? thanks, Kishore G
Re: Cutting a release 0.6.5 tonight
Thanks Lei, I fixed the first two. Here is what caused the failure. I added the code to copy all simple fields from IS to EV when we update the ExternalView in controller. This is to enable compression in ExternalView if its set in IS. I could have copied only enableCompression variable but I thought its good to have partition number/replica etc in ExternalView as well. Let me know if you foresee any problem in this. Only thing I could think of is IdealState is deleted in which case I copy the simplefields from existing externalview. I dont understand why testschedulermsg test cases are failing. Uncommenting my code does not help either. On Wed, Mar 11, 2015 at 4:06 PM, Kanak Biscuitwala kana...@hotmail.com wrote: The first two are concerning. I don't think the scheduler message test failures should block releases. From: l...@linkedin.com.INVALID To: dev@helix.apache.org CC: u...@helix.apache.org Subject: RE: Cutting a release 0.6.5 tonight Date: Wed, 11 Mar 2015 21:25:15 + The test failed are: org.apache.helix.integration.TestExternalViewUpdates.testExternalViewUpdates org.apache.helix.integration.TestEnableCompression.testEnableCompressionResource org.apache.helix.integration.TestSchedulerMessage.testSchedulerMsg3 org.apache.helix.integration.TestSchedulerMessage.testSchedulerMsg4 org.apache.helix.integration.TestSchedulerMessage.testSchedulerMsgContraints org.apache.helix.integration.TestSchedulerMessage.testSchedulerMsgUsingQueue I got these from my local build too. (mvn clean install package on helix-0.6.x) Thanks Lei -- Lei Xia Software Engineer Data Infrastructure/Distributed Data Systems/Nuage LinkedIn l...@linkedin.com www.linkedin.com/in/lxia1 From: kishore g [g.kish...@gmail.com] Sent: Wednesday, March 11, 2015 2:03 PM To: dev@helix.apache.org Cc: u...@helix.apache.org Subject: Re: Cutting a release 0.6.5 tonight Hi Lei, Can you point to the failures? thanks, Kishore G On Wed, Mar 11, 2015 at 1:19 PM, Lei Xia l...@linkedin.com.invalid wrote: Hi, Kishore I saw there are regression test failures from last two recent commits on 0.6.x branch, running from both local box and Linkedin's hudson jobs. Are we going to fix them before the release? Thanks Lei -- Lei Xia Software Engineer Data Infrastructure/Distributed Data Systems/Nuage LinkedIn l...@linkedin.com www.linkedin.com/in/lxia1 From: kishore g [g.kish...@gmail.com] Sent: Wednesday, March 11, 2015 12:04 PM To: u...@helix.apache.org Cc: dev@helix.apache.org Subject: Re: Cutting a release 0.6.5 tonight Will write a test case for migration and document the steps On Mar 11, 2015 11:37 AM, Varun Sharma va...@pinterest.com wrote: What would be the migration path from non-compressed buckets to compressed non bucket resources ? It seems even the CURRENTSTATES are being bucketed in this case, I thought that was not expected with bucketing. Does the controller read these current states appropriately ? To migrate, it seems that we would need to also rewrite the CURRENT STATES ? On Wed, Mar 11, 2015 at 10:02 AM, kishore g g.kish...@gmail.com wrote: Hi, I will work with Jason to cut a 0.6.5 release tonight. The new thing I added is to enableCompression while storing data in Zookeeper, this allows us to go up to 100k partitions per resource without having to use bucketing feature. We also fixed few bugs with bucketed resource just in case some one needs it. The property store api needs some changes, I plan to get it in today. Let me know if you need any other changes to be included. Are there any changes that went into 0.7.x branch that we need to merge it back in to 0.6.x ? thanks, Kishore G
RE: Cutting a release 0.6.5 tonight
The first two are concerning. I don't think the scheduler message test failures should block releases. From: l...@linkedin.com.INVALID To: dev@helix.apache.org CC: u...@helix.apache.org Subject: RE: Cutting a release 0.6.5 tonight Date: Wed, 11 Mar 2015 21:25:15 + The test failed are: org.apache.helix.integration.TestExternalViewUpdates.testExternalViewUpdates org.apache.helix.integration.TestEnableCompression.testEnableCompressionResource org.apache.helix.integration.TestSchedulerMessage.testSchedulerMsg3 org.apache.helix.integration.TestSchedulerMessage.testSchedulerMsg4 org.apache.helix.integration.TestSchedulerMessage.testSchedulerMsgContraints org.apache.helix.integration.TestSchedulerMessage.testSchedulerMsgUsingQueue I got these from my local build too. (mvn clean install package on helix-0.6.x) Thanks Lei -- Lei Xia Software Engineer Data Infrastructure/Distributed Data Systems/Nuage LinkedIn l...@linkedin.com www.linkedin.com/in/lxia1 From: kishore g [g.kish...@gmail.com] Sent: Wednesday, March 11, 2015 2:03 PM To: dev@helix.apache.org Cc: u...@helix.apache.org Subject: Re: Cutting a release 0.6.5 tonight Hi Lei, Can you point to the failures? thanks, Kishore G On Wed, Mar 11, 2015 at 1:19 PM, Lei Xia l...@linkedin.com.invalid wrote: Hi, Kishore I saw there are regression test failures from last two recent commits on 0.6.x branch, running from both local box and Linkedin's hudson jobs. Are we going to fix them before the release? Thanks Lei -- Lei Xia Software Engineer Data Infrastructure/Distributed Data Systems/Nuage LinkedIn l...@linkedin.com www.linkedin.com/in/lxia1 From: kishore g [g.kish...@gmail.com] Sent: Wednesday, March 11, 2015 12:04 PM To: u...@helix.apache.org Cc: dev@helix.apache.org Subject: Re: Cutting a release 0.6.5 tonight Will write a test case for migration and document the steps On Mar 11, 2015 11:37 AM, Varun Sharma va...@pinterest.com wrote: What would be the migration path from non-compressed buckets to compressed non bucket resources ? It seems even the CURRENTSTATES are being bucketed in this case, I thought that was not expected with bucketing. Does the controller read these current states appropriately ? To migrate, it seems that we would need to also rewrite the CURRENT STATES ? On Wed, Mar 11, 2015 at 10:02 AM, kishore g g.kish...@gmail.com wrote: Hi, I will work with Jason to cut a 0.6.5 release tonight. The new thing I added is to enableCompression while storing data in Zookeeper, this allows us to go up to 100k partitions per resource without having to use bucketing feature. We also fixed few bugs with bucketed resource just in case some one needs it. The property store api needs some changes, I plan to get it in today. Let me know if you need any other changes to be included. Are there any changes that went into 0.7.x branch that we need to merge it back in to 0.6.x ? thanks, Kishore G
Review Request 31975: [HELIX-573] Add support to automatically compress/uncompress data in Zookeeper
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31975/ --- Review request for helix. Bugs: HELIX-573 Repository: helix-git Description --- commit d704af4423360b382fc5f895b85ad689fa003b92 Author: Kishore Gopalakrishna g.kish...@gmail.com Date: Wed Mar 11 21:30:31 2015 -0700 [HELIX-573] Add support to automatically compress/uncompress data in Zookeeper :100644 100644 80529df... d0bdbd9... M helix-core/src/main/java/org/apache/helix/controller/stages/ExternalViewComputeStage.java :100644 100644 1552e53... 45dbd58... M helix-core/src/test/java/org/apache/helix/integration/TestEnableCompression.java Diffs - helix-core/src/main/java/org/apache/helix/controller/stages/ExternalViewComputeStage.java 80529df helix-core/src/test/java/org/apache/helix/integration/TestEnableCompression.java 1552e53 Diff: https://reviews.apache.org/r/31975/diff/ Testing --- Thanks, Kishore Gopalakrishna
Re: Review Request 31975: [HELIX-573] Add support to automatically compress/uncompress data in Zookeeper
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31975/#review76208 --- Ship it! Ship It! - Kanak Biscuitwala On March 11, 2015, 9:32 p.m., Kishore Gopalakrishna wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31975/ --- (Updated March 11, 2015, 9:32 p.m.) Review request for helix. Bugs: HELIX-573 Repository: helix-git Description --- commit d704af4423360b382fc5f895b85ad689fa003b92 Author: Kishore Gopalakrishna g.kish...@gmail.com Date: Wed Mar 11 21:30:31 2015 -0700 [HELIX-573] Add support to automatically compress/uncompress data in Zookeeper :100644 100644 80529df... d0bdbd9... M helix-core/src/main/java/org/apache/helix/controller/stages/ExternalViewComputeStage.java :100644 100644 1552e53... 45dbd58... M helix-core/src/test/java/org/apache/helix/integration/TestEnableCompression.java Diffs - helix-core/src/main/java/org/apache/helix/controller/stages/ExternalViewComputeStage.java 80529df helix-core/src/test/java/org/apache/helix/integration/TestEnableCompression.java 1552e53 Diff: https://reviews.apache.org/r/31975/diff/ Testing --- Thanks, Kishore Gopalakrishna
Re: Review Request 31970: [HELIX-574] fix bucketize resource bug in current state carryover
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31970/ --- (Updated March 12, 2015, 5:45 a.m.) Review request for helix, Kanak Biscuitwala and Kishore Gopalakrishna. Repository: helix-git Description --- [HELIX-574] fix bucketize resource bug in current state carryover One logic change is previous we don't allow drop a disabled resource. Now we change to drop a resource regardless it's disabled or not. Diffs (updated) - helix-core/src/main/java/org/apache/helix/controller/rebalancer/CustomRebalancer.java 0f3fbc4 helix-core/src/main/java/org/apache/helix/controller/rebalancer/util/ConstraintBasedAssignment.java bab357b helix-core/src/main/java/org/apache/helix/manager/zk/ParticipantManagerHelper.java 1bee2fe helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixDataAccessor.java 8c9fc8d helix-core/src/test/java/org/apache/helix/integration/TestBucketizedResource.java 2b5e2bc Diff: https://reviews.apache.org/r/31970/diff/ Testing --- Thanks, Zhen Zhang
Re: Review Request 31970: [HELIX-574] fix bucketize resource bug in current state carryover
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31970/#review76210 --- Ship it! helix-core/src/main/java/org/apache/helix/manager/zk/ParticipantManagerHelper.java https://reviews.apache.org/r/31970/#comment123697 will we ever see a partition in DROPPED state in the current state or we remove the entry after OFFLINE - DROPPED is processed helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixDataAccessor.java https://reviews.apache.org/r/31970/#comment123696 Add a comment why we are doing this - Kishore Gopalakrishna On March 12, 2015, 5:45 a.m., Zhen Zhang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31970/ --- (Updated March 12, 2015, 5:45 a.m.) Review request for helix, Kanak Biscuitwala and Kishore Gopalakrishna. Repository: helix-git Description --- [HELIX-574] fix bucketize resource bug in current state carryover One logic change is previous we don't allow drop a disabled resource. Now we change to drop a resource regardless it's disabled or not. Diffs - helix-core/src/main/java/org/apache/helix/controller/rebalancer/CustomRebalancer.java 0f3fbc4 helix-core/src/main/java/org/apache/helix/controller/rebalancer/util/ConstraintBasedAssignment.java bab357b helix-core/src/main/java/org/apache/helix/manager/zk/ParticipantManagerHelper.java 1bee2fe helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixDataAccessor.java 8c9fc8d helix-core/src/test/java/org/apache/helix/integration/TestBucketizedResource.java 2b5e2bc Diff: https://reviews.apache.org/r/31970/diff/ Testing --- Thanks, Zhen Zhang
Re: Review Request 31970: [HELIX-574] fix bucketize resource bug in current state carryover
On March 12, 2015, 5:48 a.m., Kishore Gopalakrishna wrote: helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixDataAccessor.java, line 190 https://reviews.apache.org/r/31970/diff/1/?file=891921#file891921line190 Add a comment why we are doing this will do - Zhen --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31970/#review76210 --- On March 12, 2015, 5:45 a.m., Zhen Zhang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31970/ --- (Updated March 12, 2015, 5:45 a.m.) Review request for helix, Kanak Biscuitwala and Kishore Gopalakrishna. Repository: helix-git Description --- [HELIX-574] fix bucketize resource bug in current state carryover One logic change is previous we don't allow drop a disabled resource. Now we change to drop a resource regardless it's disabled or not. Diffs - helix-core/src/main/java/org/apache/helix/controller/rebalancer/CustomRebalancer.java 0f3fbc4 helix-core/src/main/java/org/apache/helix/controller/rebalancer/util/ConstraintBasedAssignment.java bab357b helix-core/src/main/java/org/apache/helix/manager/zk/ParticipantManagerHelper.java 1bee2fe helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixDataAccessor.java 8c9fc8d helix-core/src/test/java/org/apache/helix/integration/TestBucketizedResource.java 2b5e2bc Diff: https://reviews.apache.org/r/31970/diff/ Testing --- Thanks, Zhen Zhang
[jira] [Created] (HELIX-574) fix bucketize resource bug in current state carryover
Zhen Zhang created HELIX-574: Summary: fix bucketize resource bug in current state carryover Key: HELIX-574 URL: https://issues.apache.org/jira/browse/HELIX-574 Project: Apache Helix Issue Type: Bug Reporter: Zhen Zhang -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 31970: [HELIX-574] fix bucketize resource bug in current state carryover
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31970/#review76202 --- Ship it! Ship it! (assuming tests pass) - Kanak Biscuitwala On March 11, 2015, 7:28 p.m., Zhen Zhang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31970/ --- (Updated March 11, 2015, 7:28 p.m.) Review request for helix, Kanak Biscuitwala and Kishore Gopalakrishna. Repository: helix-git Description --- [HELIX-574] fix bucketize resource bug in current state carryover One logic change is previous we don't allow drop a disabled resource. Now we change to drop a resource regardless it's disabled or not. Diffs - helix-core/src/main/java/org/apache/helix/controller/rebalancer/CustomRebalancer.java 0f3fbc4 helix-core/src/main/java/org/apache/helix/controller/rebalancer/util/ConstraintBasedAssignment.java bab357b helix-core/src/main/java/org/apache/helix/manager/zk/ParticipantManagerHelper.java 1bee2fe helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixDataAccessor.java 169c993 helix-core/src/test/java/org/apache/helix/integration/TestBucketizedResource.java 207a318 Diff: https://reviews.apache.org/r/31970/diff/ Testing --- Thanks, Zhen Zhang
