[jira] [Commented] (HBASE-13382) IntegrationTestBigLinkedList should use SecureRandom
[ https://issues.apache.org/jira/browse/HBASE-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527860#comment-14527860 ] Hudson commented on HBASE-13382: SUCCESS: Integrated in HBase-0.98 #977 (See [https://builds.apache.org/job/HBase-0.98/977/]) HBASE-13382 IntegrationTestBigLinkedList should use SecureRandom (Dima Spivak) (apurtell: rev 4e83f5781c2ce885a06f2956b803990ebadb3425) * hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java IntegrationTestBigLinkedList should use SecureRandom Key: HBASE-13382 URL: https://issues.apache.org/jira/browse/HBASE-13382 Project: HBase Issue Type: Bug Components: integration tests Reporter: Todd Lipcon Assignee: Dima Spivak Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.13 Attachments: HBASE-13382_master_v1.patch IntegrationTestBigLinkedList currently uses java.util.Random to generate its random keys. The keys are 128 bits long, but we generate them using Random.nextBytes(). The Random implementation itself only has a 48-bit seed, so even though we have a very long key string, it doesn't have anywhere near that amount of entropy. This means that after a few billion rows, it's quite likely to run into a collision: filling in a 16-byte key is equivalent to four calls to rand.nextInt(). So, for 10B rows, we are cycling through 40B different 'seed' values. With a 48-bit seed, it's quite likely we'll end up using the same seed twice, after which point any future rows generated by the colliding mappers are going to be equal. This results in broken chains and a failed verification job. The fix is simple -- we should use SecureRandom to generate the random keys, instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13382) IntegrationTestBigLinkedList should use SecureRandom
[ https://issues.apache.org/jira/browse/HBASE-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527918#comment-14527918 ] Hudson commented on HBASE-13382: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #930 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/930/]) HBASE-13382 IntegrationTestBigLinkedList should use SecureRandom (Dima Spivak) (apurtell: rev 4e83f5781c2ce885a06f2956b803990ebadb3425) * hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java IntegrationTestBigLinkedList should use SecureRandom Key: HBASE-13382 URL: https://issues.apache.org/jira/browse/HBASE-13382 Project: HBase Issue Type: Bug Components: integration tests Reporter: Todd Lipcon Assignee: Dima Spivak Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.13 Attachments: HBASE-13382_master_v1.patch IntegrationTestBigLinkedList currently uses java.util.Random to generate its random keys. The keys are 128 bits long, but we generate them using Random.nextBytes(). The Random implementation itself only has a 48-bit seed, so even though we have a very long key string, it doesn't have anywhere near that amount of entropy. This means that after a few billion rows, it's quite likely to run into a collision: filling in a 16-byte key is equivalent to four calls to rand.nextInt(). So, for 10B rows, we are cycling through 40B different 'seed' values. With a 48-bit seed, it's quite likely we'll end up using the same seed twice, after which point any future rows generated by the colliding mappers are going to be equal. This results in broken chains and a failed verification job. The fix is simple -- we should use SecureRandom to generate the random keys, instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13382) IntegrationTestBigLinkedList should use SecureRandom
[
https://issues.apache.org/jira/browse/HBASE-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395050#comment-14395050
]
Hadoop QA commented on HBASE-13382:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12709283/HBASE-13382_master_v1.patch
against master branch at commit d8b10656d00779e194c3caca118995136babce99.
ATTACHMENT ID: 12709283
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
at
org.apache.camel.component.jetty.jettyproducer.HttpJettyProducerRecipientListCustomThreadPoolTest.testRecipientList(HttpJettyProducerRecipientListCustomThreadPoolTest.java:40)
at
org.apache.camel.test.junit4.CamelTestSupport.startCamelContext(CamelTestSupport.java:477)
at
org.apache.camel.test.junit4.CamelTestSupport.doSetUp(CamelTestSupport.java:311)
at
org.apache.camel.test.junit4.CamelTestSupport.setUp(CamelTestSupport.java:217)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13556//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13556//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13556//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13556//console
This message is automatically generated.
IntegrationTestBigLinkedList should use SecureRandom
Key: HBASE-13382
URL: https://issues.apache.org/jira/browse/HBASE-13382
Project: HBase
Issue Type: Bug
Components: integration tests
Reporter: Todd Lipcon
Assignee: Dima Spivak
Attachments: HBASE-13382_master_v1.patch
IntegrationTestBigLinkedList currently uses java.util.Random to generate its
random keys. The keys are 128 bits long, but we generate them using
Random.nextBytes(). The Random implementation itself only has a 48-bit seed,
so even though we have a very long key string, it doesn't have anywhere near
that amount of entropy.
This means that after a few billion rows, it's quite likely to run into a
collision: filling in a 16-byte key is equivalent to four calls to
rand.nextInt(). So, for 10B rows, we are cycling through 40B different 'seed'
values. With a 48-bit seed, it's quite likely we'll end up using the same
seed twice, after which point any future rows generated by the colliding
mappers are going to be equal. This results in broken chains and a failed
verification job.
The fix is simple -- we should use SecureRandom to generate the random keys,
instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13382) IntegrationTestBigLinkedList should use SecureRandom
[ https://issues.apache.org/jira/browse/HBASE-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395157#comment-14395157 ] stack commented on HBASE-13382: --- The threadlocalrandom use is one-time only and the data generated is not ever read so it can be safely ignored. Let me change the Walker Random to also be secure. I'm not sure what happens if walked has a collision. Thanks [~dimaspivak] IntegrationTestBigLinkedList should use SecureRandom Key: HBASE-13382 URL: https://issues.apache.org/jira/browse/HBASE-13382 Project: HBase Issue Type: Bug Components: integration tests Reporter: Todd Lipcon Assignee: Dima Spivak Attachments: HBASE-13382_master_v1.patch IntegrationTestBigLinkedList currently uses java.util.Random to generate its random keys. The keys are 128 bits long, but we generate them using Random.nextBytes(). The Random implementation itself only has a 48-bit seed, so even though we have a very long key string, it doesn't have anywhere near that amount of entropy. This means that after a few billion rows, it's quite likely to run into a collision: filling in a 16-byte key is equivalent to four calls to rand.nextInt(). So, for 10B rows, we are cycling through 40B different 'seed' values. With a 48-bit seed, it's quite likely we'll end up using the same seed twice, after which point any future rows generated by the colliding mappers are going to be equal. This results in broken chains and a failed verification job. The fix is simple -- we should use SecureRandom to generate the random keys, instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13382) IntegrationTestBigLinkedList should use SecureRandom
[ https://issues.apache.org/jira/browse/HBASE-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395287#comment-14395287 ] Hudson commented on HBASE-13382: FAILURE: Integrated in HBase-1.0 #847 (See [https://builds.apache.org/job/HBase-1.0/847/]) HBASE-13382 IntegrationTestBigLinkedList should use SecureRandom (Dima Spivak) (stack: rev a865414472d703201e75f9f07420de92a0bfb5d6) * hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java IntegrationTestBigLinkedList should use SecureRandom Key: HBASE-13382 URL: https://issues.apache.org/jira/browse/HBASE-13382 Project: HBase Issue Type: Bug Components: integration tests Reporter: Todd Lipcon Assignee: Dima Spivak Fix For: 2.0.0, 1.0.1, 1.1.0 Attachments: HBASE-13382_master_v1.patch IntegrationTestBigLinkedList currently uses java.util.Random to generate its random keys. The keys are 128 bits long, but we generate them using Random.nextBytes(). The Random implementation itself only has a 48-bit seed, so even though we have a very long key string, it doesn't have anywhere near that amount of entropy. This means that after a few billion rows, it's quite likely to run into a collision: filling in a 16-byte key is equivalent to four calls to rand.nextInt(). So, for 10B rows, we are cycling through 40B different 'seed' values. With a 48-bit seed, it's quite likely we'll end up using the same seed twice, after which point any future rows generated by the colliding mappers are going to be equal. This results in broken chains and a failed verification job. The fix is simple -- we should use SecureRandom to generate the random keys, instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13382) IntegrationTestBigLinkedList should use SecureRandom
[ https://issues.apache.org/jira/browse/HBASE-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395395#comment-14395395 ] Hudson commented on HBASE-13382: FAILURE: Integrated in HBase-1.1 #357 (See [https://builds.apache.org/job/HBase-1.1/357/]) HBASE-13382 IntegrationTestBigLinkedList should use SecureRandom (Dima Spivak) (stack: rev e3d6cdd072d153f3d23fc59e6326abf182353bb8) * hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java IntegrationTestBigLinkedList should use SecureRandom Key: HBASE-13382 URL: https://issues.apache.org/jira/browse/HBASE-13382 Project: HBase Issue Type: Bug Components: integration tests Reporter: Todd Lipcon Assignee: Dima Spivak Fix For: 2.0.0, 1.0.1, 1.1.0 Attachments: HBASE-13382_master_v1.patch IntegrationTestBigLinkedList currently uses java.util.Random to generate its random keys. The keys are 128 bits long, but we generate them using Random.nextBytes(). The Random implementation itself only has a 48-bit seed, so even though we have a very long key string, it doesn't have anywhere near that amount of entropy. This means that after a few billion rows, it's quite likely to run into a collision: filling in a 16-byte key is equivalent to four calls to rand.nextInt(). So, for 10B rows, we are cycling through 40B different 'seed' values. With a 48-bit seed, it's quite likely we'll end up using the same seed twice, after which point any future rows generated by the colliding mappers are going to be equal. This results in broken chains and a failed verification job. The fix is simple -- we should use SecureRandom to generate the random keys, instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13382) IntegrationTestBigLinkedList should use SecureRandom
[ https://issues.apache.org/jira/browse/HBASE-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395482#comment-14395482 ] Hudson commented on HBASE-13382: SUCCESS: Integrated in HBase-TRUNK #6343 (See [https://builds.apache.org/job/HBase-TRUNK/6343/]) HBASE-13382 IntegrationTestBigLinkedList should use SecureRandom (Dima Spivak) (stack: rev f561ef71a505d9e3d63a9ac4c1a950d7c066dc78) * hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java IntegrationTestBigLinkedList should use SecureRandom Key: HBASE-13382 URL: https://issues.apache.org/jira/browse/HBASE-13382 Project: HBase Issue Type: Bug Components: integration tests Reporter: Todd Lipcon Assignee: Dima Spivak Fix For: 2.0.0, 1.0.1, 1.1.0 Attachments: HBASE-13382_master_v1.patch IntegrationTestBigLinkedList currently uses java.util.Random to generate its random keys. The keys are 128 bits long, but we generate them using Random.nextBytes(). The Random implementation itself only has a 48-bit seed, so even though we have a very long key string, it doesn't have anywhere near that amount of entropy. This means that after a few billion rows, it's quite likely to run into a collision: filling in a 16-byte key is equivalent to four calls to rand.nextInt(). So, for 10B rows, we are cycling through 40B different 'seed' values. With a 48-bit seed, it's quite likely we'll end up using the same seed twice, after which point any future rows generated by the colliding mappers are going to be equal. This results in broken chains and a failed verification job. The fix is simple -- we should use SecureRandom to generate the random keys, instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13382) IntegrationTestBigLinkedList should use SecureRandom
[ https://issues.apache.org/jira/browse/HBASE-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14393364#comment-14393364 ] stack commented on HBASE-13382: --- Hopefully this is cause of infinite loop in HBASE-13161 IntegrationTestBigLinkedList should use SecureRandom Key: HBASE-13382 URL: https://issues.apache.org/jira/browse/HBASE-13382 Project: HBase Issue Type: Bug Components: integration tests Reporter: Todd Lipcon IntegrationTestBigLinkedList currently uses java.util.Random to generate its random keys. The keys are 128 bits long, but we generate them using Random.nextBytes(). The Random implementation itself only has a 48-bit seed, so even though we have a very long key string, it doesn't have anywhere near that amount of entropy. This means that after a few billion rows, it's quite likely to run into a collision: filling in a 16-byte key is equivalent to four calls to rand.nextInt(). So, for 10B rows, we are cycling through 40B different 'seed' values. With a 48-bit seed, it's quite likely we'll end up using the same seed twice, after which point any future rows generated by the colliding mappers are going to be equal. This results in broken chains and a failed verification job. The fix is simple -- we should use SecureRandom to generate the random keys, instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13382) IntegrationTestBigLinkedList should use SecureRandom
[ https://issues.apache.org/jira/browse/HBASE-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14393323#comment-14393323 ] Dima Spivak commented on HBASE-13382: - Want me to take this, [~tlipcon]? Looks like a one line fix. IntegrationTestBigLinkedList should use SecureRandom Key: HBASE-13382 URL: https://issues.apache.org/jira/browse/HBASE-13382 Project: HBase Issue Type: Bug Components: integration tests Reporter: Todd Lipcon IntegrationTestBigLinkedList currently uses java.util.Random to generate its random keys. The keys are 128 bits long, but we generate them using Random.nextBytes(). The Random implementation itself only has a 48-bit seed, so even though we have a very long key string, it doesn't have anywhere near that amount of entropy. This means that after a few billion rows, it's quite likely to run into a collision: filling in a 16-byte key is equivalent to four calls to rand.nextInt(). So, for 10B rows, we are cycling through 40B different 'seed' values. With a 48-bit seed, it's quite likely we'll end up using the same seed twice, after which point any future rows generated by the colliding mappers are going to be equal. This results in broken chains and a failed verification job. The fix is simple -- we should use SecureRandom to generate the random keys, instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13382) IntegrationTestBigLinkedList should use SecureRandom
[ https://issues.apache.org/jira/browse/HBASE-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14393769#comment-14393769 ] Enis Soztutar commented on HBASE-13382: --- Related: HBASE-12891. IntegrationTestBigLinkedList should use SecureRandom Key: HBASE-13382 URL: https://issues.apache.org/jira/browse/HBASE-13382 Project: HBase Issue Type: Bug Components: integration tests Reporter: Todd Lipcon Assignee: Dima Spivak IntegrationTestBigLinkedList currently uses java.util.Random to generate its random keys. The keys are 128 bits long, but we generate them using Random.nextBytes(). The Random implementation itself only has a 48-bit seed, so even though we have a very long key string, it doesn't have anywhere near that amount of entropy. This means that after a few billion rows, it's quite likely to run into a collision: filling in a 16-byte key is equivalent to four calls to rand.nextInt(). So, for 10B rows, we are cycling through 40B different 'seed' values. With a 48-bit seed, it's quite likely we'll end up using the same seed twice, after which point any future rows generated by the colliding mappers are going to be equal. This results in broken chains and a failed verification job. The fix is simple -- we should use SecureRandom to generate the random keys, instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
