[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16523748#comment-16523748 ] Hudson commented on HBASE-20403: Results for branch master [build #377 on builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/377/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://builds.apache.org/job/HBase%20Nightly/job/master/377//General_Nightly_Build_Report/] (/) {color:green}+1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://builds.apache.org/job/HBase%20Nightly/job/master/377//JDK8_Nightly_Build_Report_(Hadoop2)/] (x) {color:red}-1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://builds.apache.org/job/HBase%20Nightly/job/master/377//JDK8_Nightly_Build_Report_(Hadoop3)/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Assignee: Todd Lipcon >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.0.2 > > Attachments: hbase-20403.patch, hbase-20403.patch > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be: > * if file is encrypted with FileStatus#isEncrypted() and do not prefetch. > * document that hbase.rs.prefetchblocksonopen cannot be true if file is > encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16523216#comment-16523216 ] Hudson commented on HBASE-20403: Results for branch branch-2.0 [build #474 on builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/474/]: (/) *{color:green}+1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/474//General_Nightly_Build_Report/] (/) {color:green}+1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/474//JDK8_Nightly_Build_Report_(Hadoop2)/] (/) {color:green}+1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/474//JDK8_Nightly_Build_Report_(Hadoop3)/] (/) {color:green}+1 source release artifact{color} -- See build output for details. > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Assignee: Todd Lipcon >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.0.2 > > Attachments: hbase-20403.patch, hbase-20403.patch > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be: > * if file is encrypted with FileStatus#isEncrypted() and do not prefetch. > * document that hbase.rs.prefetchblocksonopen cannot be true if file is > encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16523193#comment-16523193 ] Hudson commented on HBASE-20403: Results for branch branch-2 [build #908 on builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/908/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/908//General_Nightly_Build_Report/] (x) {color:red}-1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/908//JDK8_Nightly_Build_Report_(Hadoop2)/] (x) {color:red}-1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/908//JDK8_Nightly_Build_Report_(Hadoop3)/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Assignee: Todd Lipcon >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.0.2 > > Attachments: hbase-20403.patch, hbase-20403.patch > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be: > * if file is encrypted with FileStatus#isEncrypted() and do not prefetch. > * document that hbase.rs.prefetchblocksonopen cannot be true if file is > encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16522743#comment-16522743 ] Andrew Purtell commented on HBASE-20403: I don't believe branch-1 is affected because of locking in hfileblock. > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Assignee: Todd Lipcon >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.0.2 > > Attachments: hbase-20403.patch, hbase-20403.patch > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be: > * if file is encrypted with FileStatus#isEncrypted() and do not prefetch. > * document that hbase.rs.prefetchblocksonopen cannot be true if file is > encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16522725#comment-16522725 ] Todd Lipcon commented on HBASE-20403: - I would guess it's not affected because it has locking in the file reader path. The locking was removed by HBASE-17917 in 2.0 > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Assignee: Todd Lipcon >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.0.2 > > Attachments: hbase-20403.patch, hbase-20403.patch > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be: > * if file is encrypted with FileStatus#isEncrypted() and do not prefetch. > * document that hbase.rs.prefetchblocksonopen cannot be true if file is > encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16522721#comment-16522721 ] Mike Drob commented on HBASE-20403: --- is branch-1 affected by this also? I'd imagine yes since I don't think the scan code has been rewritten, but I also would have expected to see this before if so. > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Assignee: Todd Lipcon >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.0.2 > > Attachments: hbase-20403.patch, hbase-20403.patch > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be: > * if file is encrypted with FileStatus#isEncrypted() and do not prefetch. > * document that hbase.rs.prefetchblocksonopen cannot be true if file is > encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16522717#comment-16522717 ] Andrew Purtell commented on HBASE-20403: Thanks for the fix and the commit [~tlipcon] > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Assignee: Todd Lipcon >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.0.2 > > Attachments: hbase-20403.patch, hbase-20403.patch > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be: > * if file is encrypted with FileStatus#isEncrypted() and do not prefetch. > * document that hbase.rs.prefetchblocksonopen cannot be true if file is > encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16522693#comment-16522693 ] Mike Drob commented on HBASE-20403: --- bq. Got a report back from an internal test cluster who was previously reproducing this issue. With this patch applied the issue seems to be resolved. Sounds good, you still have commit access, right? ;) +1 > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Assignee: Todd Lipcon >Priority: Major > Fix For: 3.0.0 > > Attachments: hbase-20403.patch, hbase-20403.patch > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be: > * if file is encrypted with FileStatus#isEncrypted() and do not prefetch. > * document that hbase.rs.prefetchblocksonopen cannot be true if file is > encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16522682#comment-16522682 ] Todd Lipcon commented on HBASE-20403: - Got a report back from an internal test cluster who was previously reproducing this issue. With this patch applied the issue seems to be resolved. > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Assignee: Todd Lipcon >Priority: Major > Fix For: 3.0.0 > > Attachments: hbase-20403.patch, hbase-20403.patch > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be: > * if file is encrypted with FileStatus#isEncrypted() and do not prefetch. > * document that hbase.rs.prefetchblocksonopen cannot be true if file is > encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16520778#comment-16520778 ] Ted Yu commented on HBASE-20403: +1 > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Assignee: Todd Lipcon >Priority: Major > Fix For: 3.0.0 > > Attachments: hbase-20403.patch, hbase-20403.patch > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be: > * if file is encrypted with FileStatus#isEncrypted() and do not prefetch. > * document that hbase.rs.prefetchblocksonopen cannot be true if file is > encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16520694#comment-16520694 ] Todd Lipcon commented on HBASE-20403: - Sure, I filed HADOOP-15557 > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Assignee: Todd Lipcon >Priority: Major > Fix For: 3.0.0 > > Attachments: hbase-20403.patch, hbase-20403.patch > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be: > * if file is encrypted with FileStatus#isEncrypted() and do not prefetch. > * document that hbase.rs.prefetchblocksonopen cannot be true if file is > encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16520629#comment-16520629 ] Mike Drob commented on HBASE-20403: --- bq. I think on the HDFS side (Hadoop common, really), we should add some sanity checking to prevent concurrent use and throw ConcurrentModificationException when we detect it so such bugs are obvious in the future instead of being very difficult to diagnose. Can you file a Hadoop Common JIRA for this with some detail and link it here? Thanks Todd! > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Assignee: Todd Lipcon >Priority: Major > Fix For: 3.0.0 > > Attachments: hbase-20403.patch, hbase-20403.patch > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be: > * if file is encrypted with FileStatus#isEncrypted() and do not prefetch. > * document that hbase.rs.prefetchblocksonopen cannot be true if file is > encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519982#comment-16519982 ] Hadoop QA commented on HBASE-20403: --- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 1s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 49s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 11s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 31s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 36s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 10m 24s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green}132m 48s{color} | {color:green} hbase-server in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}175m 8s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b | | JIRA Issue | HBASE-20403 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12928693/hbase-20403.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 8b6974dfd586 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh | | git revision | master / bc9f9ae080 | | maven | version: Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) | | Default Java | 1.8.0_171 | | findbugs | v3.1.0-RC3 | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/13338/testReport/ | | Max. process+thread count | 4328 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/13338/console | | Powered by | Apache Yetus 0.7.0 http://yetus.apache.org | This message was automatically generated. > Prefetch sometimes doesn
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519874#comment-16519874 ] Todd Lipcon commented on HBASE-20403: - OK. New revision fixes the checkstyle. If someone out there knows how to reproduce the originally-reported issue and can check that this patch fixes it, that would be great confirmation that there isn't another issue lurking. > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Assignee: Todd Lipcon >Priority: Major > Fix For: 3.0.0 > > Attachments: hbase-20403.patch, hbase-20403.patch > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be: > * if file is encrypted with FileStatus#isEncrypted() and do not prefetch. > * document that hbase.rs.prefetchblocksonopen cannot be true if file is > encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519750#comment-16519750 ] Andrew Purtell commented on HBASE-20403: It should be ok for prefetch to be slower. It is meant as a performance optimization, for prepopulating the block cache. At preload time we don't know which if any blocks are going to be immediately required. > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Assignee: Todd Lipcon >Priority: Major > Fix For: 3.0.0 > > Attachments: hbase-20403.patch > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be: > * if file is encrypted with FileStatus#isEncrypted() and do not prefetch. > * document that hbase.rs.prefetchblocksonopen cannot be true if file is > encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518879#comment-16518879 ] Todd Lipcon commented on HBASE-20403: - bq. even though the underlying, wrapped DFSInputStream seems mostly thread-safe That's an interesting point. I just looked at DFSInputStream and sure enough these non-positional methods are marked synchronized. However, it's somewhat odd because you'd still need some external synchronization to know where you're reading from. That is to say, if one thread is doing a 'seek, then read' at the same time as the other, they could interleave and one thread reads from the other thread's position. That said, I could see the synchronization of DFSInputStream hiding bugs -- maybe it happens that sometimes one thread reads the data meant for another thread and just proceeds having read the wrong block. Or, it reads the wrong data because of the race, sees it as an HBase-level checksum failure, and performs a retry. In the Crypto case, because the input stream is not synchronized, it now ends up in a crash or an odd exception instead of "just reading the wrong data". I think on the HDFS side (Hadoop common, really), we should add some sanity checking to prevent concurrent use and throw ConcurrentModificationException when we detect it so such bugs are obvious in the future instead of being very difficult to diagnose. > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Assignee: Todd Lipcon >Priority: Major > Fix For: 3.0.0 > > Attachments: hbase-20403.patch > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be: > * if file is encrypted with FileStatus#isEncrypted() and do not prefetch. > * document that hbase.rs.prefetchblocksonopen cannot be true if file is > encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518787#comment-16518787 ] Wei-Chiu Chuang commented on HBASE-20403: - Excellent Todd! Also note that the Hadoop CryptoInputStream is not meant to be thread-safe, even though the underlying, wrapped DFSInputStream seems mostly thread-safe. That plus HBase prefetch threads basically makes data race more severe. {code:java|title=CryptoInputStream.java} /** * CryptoInputStream decrypts data. It is not thread-safe. AES CTR mode is * required in order to ensure that the plain text and cipher text have a 1:1 * mapping. The decryption is buffer based. The key points of the decryption * are (1) calculating the counter and (2) padding through stream position: * * counter = base + pos/(algorithm blocksize); * padding = pos%(algorithm blocksize); * * The underlying stream offset is maintained as state. */ @InterfaceAudience.Private @InterfaceStability.Evolving public class CryptoInputStream extends FilterInputStream implements Seekable, PositionedReadable, ByteBufferReadable, HasFileDescriptor, CanSetDropBehind, CanSetReadahead, HasEnhancedByteBufferAccess, ReadableByteChannel, CanUnbuffer, StreamCapabilities { {code} > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Assignee: Todd Lipcon >Priority: Major > Fix For: 3.0.0 > > Attachments: hbase-20403.patch > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be: > * if file is encrypted with FileStatus#isEncrypted() and do not prefetch. > * document that hbase.rs.prefetchblocksonopen cannot be true if file is > encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518768#comment-16518768 ] Hadoop QA commented on HBASE-20403: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m 32s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 30s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 37s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 57s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 2s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 47s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 35s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 57s{color} | {color:red} hbase-server: The patch generated 1 new + 54 unchanged - 0 fixed = 55 total (was 54) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 1s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 8m 54s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green}109m 45s{color} | {color:green} hbase-server in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 22s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}148m 31s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b | | JIRA Issue | HBASE-20403 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12928543/hbase-20403.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 6ff5bdee580e 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 72784c2d83 | | maven | version: Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) | | Default Java | 1.8.0_171 | | findbugs | v3.1.0-RC3 | | checkstyle | https://builds.apache.org/job/PreCommit-HBASE-Build/13323/artifact/patchprocess/diff-checkstyle-hbase-server.txt | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/13323/testReport/ | | Max. process+thread count | 4667 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.a
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518684#comment-16518684 ] Todd Lipcon commented on HBASE-20403: - I'm guessing maybe this is partially due to HBASE-17917. It seems there used to be a lock {{streamLock}} inside HFileBlock which prevented multiple threads from doing non-positional reads on top of each other. That patch basically removed usage of this lock except for in "unbuffer", but now it no longer really protects anything (except perhaps multiple unbuffers from each other?). I'm not sure if there are other cases where non-preads are used but worth auditing to make sure my patch here isn't too shallow. Also worth noting that moving from streaming to positional probably means that the prefetching will be slower. > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Assignee: Todd Lipcon >Priority: Major > Fix For: 3.0.0 > > Attachments: hbase-20403.patch > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be: > * if file is encrypted with FileStatus#isEncrypted() and do not prefetch. > * document that hbase.rs.prefetchblocksonopen cannot be true if file is > encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518682#comment-16518682 ] Umesh Agashe commented on HBASE-20403: -- +1 for the patch! Nice. Thanks [~tlipcon]! > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Assignee: Todd Lipcon >Priority: Major > Fix For: 3.0.0 > > Attachments: hbase-20403.patch > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be: > * if file is encrypted with FileStatus#isEncrypted() and do not prefetch. > * document that hbase.rs.prefetchblocksonopen cannot be true if file is > encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518681#comment-16518681 ] stack commented on HBASE-20403: --- Thanks [~tlipcon]. Nice. Makes sense (especially given you can repro). This code as-is has been in place for ages too. > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Assignee: Todd Lipcon >Priority: Major > Fix For: 3.0.0 > > Attachments: hbase-20403.patch > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be: > * if file is encrypted with FileStatus#isEncrypted() and do not prefetch. > * document that hbase.rs.prefetchblocksonopen cannot be true if file is > encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16517805#comment-16517805 ] Todd Lipcon commented on HBASE-20403: - Hello from the peanut gallery! Looking at the implementation of prefetch, it seems like the prefetch task scheduled on a separate thread calls readBlock() on the HFileReaderImpl even though there might be concurrent calls from the main (scanner) thread. It calls readBlock() with pread == false, which means that it ends up screwing with the file position, buffers, and underlying codec from the main thread. Seems like that could easily cause invalid data reads, weird buffer offsets, and crypto library crashes (due to concurrent usage of the same cipher). Am I mis-remembering the thread safety guarantees of HFileReader? I had thought it was not meant to be thread-safe, but the prefetching is basically multi-threaded access to a single instance. > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Major > Fix For: 3.0.0 > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be: > * if file is encrypted with FileStatus#isEncrypted() and do not prefetch. > * document that hbase.rs.prefetchblocksonopen cannot be true if file is > encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510447#comment-16510447 ] Wei-Chiu Chuang commented on HBASE-20403: - This bug seems to have different facets: {noformat} 2018-06-12 10:35:45,613 WARN org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180606004324/1fc1713dc72a0d3351b3214add699e59/meta/543bace1c8f545a5a34d8f1dc6be6063, offset=0, end=153035 java.lang.NegativeArraySizeException at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1768) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-06-12 10:35:45,614 WARN org.apache.hadoop.hbase.io.hfile.HFile: HBase checksum verification failed for file hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180606004324/1fc1713dc72a0d3351b3214add699e59/meta/543bace1c8f545a5a34d8f1dc6be6063 at offset 0 filesize 157916. Retrying read with HDFS checksums turned on... 2018-06-12 10:35:45,626 WARN org.apache.hadoop.hbase.io.hfile.HFile: HDFS checksum verification succeeded for file hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180606004324/1fc1713dc72a0d3351b3214add699e59/meta/543bace1c8f545a5a34d8f1dc6be6063 at offset 0 filesize 157916 {noformat} > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Major > Fix For: 3.0.0 > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be:
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16443242#comment-16443242 ] stack commented on HBASE-20403: --- Hmm. That makes sense [~jojochuang] and [~apurtell]. In our cache we have stuff like next blocks header because it is small and in a few cases we can just read the block data and save a separate read of the block metadata. I had conjured a scenario where in the rare case where we got to use the header length from cache -- a totally cached file has a block evicted and then a new client comes in to do a full scan -- that the pread into the encrypted block landed us in "random location". The suggestion that the stream should be taking care of the positional math even when encrypted-at-rest sounds right. My bad. I suppose next thing here is to figure a way of reproducing the issue so we can study where we are going awry. > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Major > Fix For: 2.0.0 > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be: > * if file is encrypted with FileStatus#isEncrypted() and do not prefetch. > * document that hbase.rs.prefetchblocksonopen cannot be true if file is > encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16439961#comment-16439961 ] Andrew Purtell commented on HBASE-20403: AFAIK, prefetch does a pass over all blocks of the hfile, reading in index and data blocks, in a manner similar to HFileReader but not 100% reusing reader code for the purpose. Maybe a refactor would help. Maybe the reader was updated for some reason but the prefetch code not. It's quite unlikely prefetch code has been exercised as well. > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Priority: Major > Fix For: 2.0.0 > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be: > * if file is encrypted with FileStatus#isEncrypted() and do not prefetch. > * document that hbase.rs.prefetchblocksonopen cannot be true if file is > encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16439957#comment-16439957 ] Andrew Purtell commented on HBASE-20403: What about prefetch triggers this specifically? > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Priority: Major > Fix For: 2.0.0 > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be: > * if file is encrypted with FileStatus#isEncrypted() and do not prefetch. > * document that hbase.rs.prefetchblocksonopen cannot be true if file is > encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16439949#comment-16439949 ] Umesh Agashe commented on HBASE-20403: -- Out of 4 ITBLL runs, this stack showed up in log files for 3 runs. 4 runs after disabling prefetch pass and no logs have the above stack trace. > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Priority: Major > Fix For: 2.0.0 > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be: > * if file is encrypted with FileStatus#isEncrypted() and do not prefetch. > * document that hbase.rs.prefetchblocksonopen cannot be true if file is > encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438210#comment-16438210 ] Andrew Purtell commented on HBASE-20403: I came to also suggest it's an HDFS bug, and edge case, and wouldn't this also affect other reads besides the prefetch case? I don't follow what you mean about header sizes. We see our stream of bytes at the requested offset regardless of HDFS metadata or internals. I think it may be a rarely occurring seek bug when HDFS is using its CryptoInputStream > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Priority: Major > Fix For: 2.0.0 > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be: > * if file is encrypted with FileStatus#isEncrypted() and do not prefetch. > * document that hbase.rs.prefetchblocksonopen cannot be true if file is > encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16436397#comment-16436397 ] Umesh Agashe commented on HBASE-20403: -- Its possible. AFAIK, there is an optimization in the HBase code where we try to optimize an extra seek by reading next block's header while reading the current block. As header size when encrypted/ not encrypted may be different, its possible that HBase is trying to read more than the buffer size. > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Priority: Major > Fix For: 2.0.0 > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be: > * if file is encrypted with FileStatus#isEncrypted() and do not prefetch. > * document that hbase.rs.prefetchblocksonopen cannot be true if file is > encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[ https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16436334#comment-16436334 ] Wei-Chiu Chuang commented on HBASE-20403: - This looks like a HDFS bug more than a HBASE one. CryptoInputStream is supposed to provide an abstraction such that caller shouldn't care what's the actual offset should be. > Prefetch sometimes doesn't work with encrypted file system > -- > > Key: HBASE-20403 > URL: https://issues.apache.org/jira/browse/HBASE-20403 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Priority: Major > Fix For: 2.0.0 > > > Log from long running test has following stack trace a few times: > {code} > 2018-04-09 18:33:21,523 WARN > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch > path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988, > offset=36884200, end=231005989 > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:275) > at > org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183) > at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Size on disk calculations seem to get messed up due to encryption. Possible > fixes can be: > * if file is encrypted with FileStatus#isEncrypted() and do not prefetch. > * document that hbase.rs.prefetchblocksonopen cannot be true if file is > encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)