[jira] [Commented] (HBASE-15362) Compression Algorithm does not respect config params from hbase-site
[ https://issues.apache.org/jira/browse/HBASE-15362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15329187#comment-15329187 ] Anoop Sam John commented on HBASE-15362: [~saint@gmail.com] pls note my reply above bq.It is the same conf object which got created at constructor, is getting passed in getCodec() call. But ya as u said, it takes some time to understand the object flow and usage. Can we get this direct fix in? I think the fix as such make sense. The ugly code flow may be we need to check. Am +1 for commiting this. > Compression Algorithm does not respect config params from hbase-site > > > Key: HBASE-15362 > URL: https://issues.apache.org/jira/browse/HBASE-15362 > Project: HBase > Issue Type: Sub-task >Reporter: deepankar >Assignee: deepankar >Priority: Trivial > Attachments: HBASE-15362.patch > > > Compression creates conf using new Configuration() and this leads to it not > respecting the confs set in hbase-site, fixing it is trivial using > HBaseConfiguration.create() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15362) Compression Algorithm does not respect config params from hbase-site
[
https://issues.apache.org/jira/browse/HBASE-15362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292973#comment-15292973
]
Hadoop QA commented on HBASE-15362:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 55s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.1 2.5.2 2.6.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 5s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 32s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12790583/HBASE-15362.patch |
| JIRA Issue | HBASE-15362 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux proserpina.apache.org 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/test_framework/yetus-0.2.1/lib/precommit/personality/hbase.sh
|
| git revision | master / af5146e |
| Default Java | 1.7.0_79 |
| Multi-JDK versions | /home/jenkins/tools/java/jdk1.8.0:1.8.0
/usr/local/jenkins/java/jdk1.7.0_79:1.7.0_79 |
| findbugs | v3.0.0 |
| Test Results
[jira] [Commented] (HBASE-15362) Compression Algorithm does not respect config params from hbase-site
[ https://issues.apache.org/jira/browse/HBASE-15362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292854#comment-15292854 ] Anoop Sam John commented on HBASE-15362: It is the same conf object which got created at constructor, is getting passed in getCodec() call. > Compression Algorithm does not respect config params from hbase-site > > > Key: HBASE-15362 > URL: https://issues.apache.org/jira/browse/HBASE-15362 > Project: HBase > Issue Type: Sub-task >Reporter: deepankar >Assignee: deepankar >Priority: Trivial > Attachments: HBASE-15362.patch > > > Compression creates conf using new Configuration() and this leads to it not > respecting the confs set in hbase-site, fixing it is trivial using > HBaseConfiguration.create() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15362) Compression Algorithm does not respect config params from hbase-site
[ https://issues.apache.org/jira/browse/HBASE-15362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216605#comment-15216605 ] deepankar commented on HBASE-15362: --- Sure will create a jira > Compression Algorithm does not respect config params from hbase-site > > > Key: HBASE-15362 > URL: https://issues.apache.org/jira/browse/HBASE-15362 > Project: HBase > Issue Type: Sub-task >Reporter: deepankar >Assignee: deepankar >Priority: Trivial > Attachments: HBASE-15362.patch > > > Compression creates conf using new Configuration() and this leads to it not > respecting the confs set in hbase-site, fixing it is trivial using > HBaseConfiguration.create() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15362) Compression Algorithm does not respect config params from hbase-site
[ https://issues.apache.org/jira/browse/HBASE-15362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216050#comment-15216050 ] stack commented on HBASE-15362: --- On the patch, I think the original code is wrong where it creates a Configuration on construction of an Algorithm and then takes a Configuration when you ask for a Codec. We should get rid of the first instance. Would be good too if we could avoid mention of explicit HBaseConfiguration in here so the Compressor is more portable Need to spend more than five minutes looking at this. Thanks [~dvdreddy] > Compression Algorithm does not respect config params from hbase-site > > > Key: HBASE-15362 > URL: https://issues.apache.org/jira/browse/HBASE-15362 > Project: HBase > Issue Type: Bug >Reporter: deepankar >Assignee: deepankar >Priority: Trivial > Attachments: HBASE-15362.patch > > > Compression creates conf using new Configuration() and this leads to it not > respecting the confs set in hbase-site, fixing it is trivial using > HBaseConfiguration.create() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15362) Compression Algorithm does not respect config params from hbase-site
[ https://issues.apache.org/jira/browse/HBASE-15362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216025#comment-15216025 ] stack commented on HBASE-15362: --- Please raise your observation that we may be able to do w/o the BlockDecompressorStream as a subtask of HBASE-1 > Compression Algorithm does not respect config params from hbase-site > > > Key: HBASE-15362 > URL: https://issues.apache.org/jira/browse/HBASE-15362 > Project: HBase > Issue Type: Bug >Reporter: deepankar >Assignee: deepankar >Priority: Trivial > Attachments: HBASE-15362.patch > > > Compression creates conf using new Configuration() and this leads to it not > respecting the confs set in hbase-site, fixing it is trivial using > HBaseConfiguration.create() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15362) Compression Algorithm does not respect config params from hbase-site
[ https://issues.apache.org/jira/browse/HBASE-15362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15215056#comment-15215056 ] deepankar commented on HBASE-15362: --- But this change is very specific to LZO and specialized (hacky) , is it ok include it ? We hacked it as a temporary solution. > Compression Algorithm does not respect config params from hbase-site > > > Key: HBASE-15362 > URL: https://issues.apache.org/jira/browse/HBASE-15362 > Project: HBase > Issue Type: Bug >Reporter: deepankar >Assignee: deepankar >Priority: Trivial > Attachments: HBASE-15362.patch > > > Compression creates conf using new Configuration() and this leads to it not > respecting the confs set in hbase-site, fixing it is trivial using > HBaseConfiguration.create() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15362) Compression Algorithm does not respect config params from hbase-site
[ https://issues.apache.org/jira/browse/HBASE-15362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15215052#comment-15215052 ] Ted Yu commented on HBASE-15362: Do you want to include the above change in a separate JIRA ? > Compression Algorithm does not respect config params from hbase-site > > > Key: HBASE-15362 > URL: https://issues.apache.org/jira/browse/HBASE-15362 > Project: HBase > Issue Type: Bug >Reporter: deepankar >Assignee: deepankar >Priority: Trivial > Attachments: HBASE-15362.patch > > > Compression creates conf using new Configuration() and this leads to it not > respecting the confs set in hbase-site, fixing it is trivial using > HBaseConfiguration.create() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15362) Compression Algorithm does not respect config params from hbase-site
[ https://issues.apache.org/jira/browse/HBASE-15362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15215047#comment-15215047 ] deepankar commented on HBASE-15362: --- Depends on the default block size we have a default block size of 16 kb and this allocates a buffer of 64 kb , so in per read we are saving 64 - 16 kb of buffer I think. > Compression Algorithm does not respect config params from hbase-site > > > Key: HBASE-15362 > URL: https://issues.apache.org/jira/browse/HBASE-15362 > Project: HBase > Issue Type: Bug >Reporter: deepankar >Assignee: deepankar >Priority: Trivial > Attachments: HBASE-15362.patch > > > Compression creates conf using new Configuration() and this leads to it not > respecting the confs set in hbase-site, fixing it is trivial using > HBaseConfiguration.create() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15362) Compression Algorithm does not respect config params from hbase-site
[ https://issues.apache.org/jira/browse/HBASE-15362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15215010#comment-15215010 ] Ted Yu commented on HBASE-15362: [~dvdreddy]: How much reduction in garbage did you observe with the above change ? Thanks > Compression Algorithm does not respect config params from hbase-site > > > Key: HBASE-15362 > URL: https://issues.apache.org/jira/browse/HBASE-15362 > Project: HBase > Issue Type: Bug >Reporter: deepankar >Assignee: deepankar >Priority: Trivial > Attachments: HBASE-15362.patch > > > Compression creates conf using new Configuration() and this leads to it not > respecting the confs set in hbase-site, fixing it is trivial using > HBaseConfiguration.create() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15362) Compression Algorithm does not respect config params from hbase-site
[
https://issues.apache.org/jira/browse/HBASE-15362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174292#comment-15174292
]
deepankar commented on HBASE-15362:
---
I was trying to set {{io.compression.codec.lzo.buffersize}}, the reason for
setting this is to decrease the buffer size allocated in
BlockDecompressorStream, it defaults to 64 kb and if you are missing cache
frequently then this was causing some significant amount of garbage.
Since you brought this up does it make sense that we should recognize that some
(almost all in the current case) of the codecs does not need to create the
above stream and directly send the calls to codec or the Decompressor class
exposed by the codec, it will avoid this 64 kb allocations across the board and
the decompressors are recycled so the buffers inside them are not a problem.
Emulating the code inside the BlockDecompressorStream, the only problem is it
cannot work with byte arrays, we special cased this internal and the code
turned out be very small, pasting this part here
{code}
// Code similar to the code in BlockDecompressionStream#decompress
Decompressor decompressor = null;
try {
decompressor = compressAlgo.getDecompressor();
int curInputOffset = inputOffset, curOutputOffset = destOffset;
int originalBlockSize = Bytes.toInt(inputArray, curInputOffset, 4);
int noUncompressedBytes = 0;
curInputOffset += 4;
// Iterate until you finish the input
while (curInputOffset < (inputOffset + compressedSize)
&& noUncompressedBytes < uncompressedSize) {
if (decompressor.needsInput()) {
int blockSize = Bytes.toInt(inputArray, curInputOffset, 4);
curInputOffset += 4;
decompressor.setInput(inputArray, curInputOffset, blockSize);
curInputOffset += blockSize;
}
int n = decompressor.decompress(dest, curOutputOffset,
uncompressedSize - noUncompressedBytes);
noUncompressedBytes += n;
curOutputOffset += n;
if (n == 0) {
if (decompressor.finished() || decompressor.needsDictionary()) {
if (noUncompressedBytes >= originalBlockSize) {
throw new IOException("uncompressed bytes >= orginal size");
}
}
}
}
if (noUncompressedBytes != uncompressedSize) {
throw new IOException("Premature EOF from the input bytes");
}
} finally {
if (decompressor != null) {
compressAlgo.returnDecompressor(decompressor);
}
}
{code}
> Compression Algorithm does not respect config params from hbase-site
>
>
> Key: HBASE-15362
> URL: https://issues.apache.org/jira/browse/HBASE-15362
> Project: HBase
> Issue Type: Bug
>Reporter: deepankar
>Assignee: deepankar
>Priority: Trivial
> Attachments: HBASE-15362.patch
>
>
> Compression creates conf using new Configuration() and this leads to it not
> respecting the confs set in hbase-site, fixing it is trivial using
> HBaseConfiguration.create()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15362) Compression Algorithm does not respect config params from hbase-site
[ https://issues.apache.org/jira/browse/HBASE-15362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173516#comment-15173516 ] Anoop Sam John commented on HBASE-15362: So which all config params in hbase xml, the compression algos need to use? > Compression Algorithm does not respect config params from hbase-site > > > Key: HBASE-15362 > URL: https://issues.apache.org/jira/browse/HBASE-15362 > Project: HBase > Issue Type: Bug >Reporter: deepankar >Assignee: deepankar >Priority: Trivial > Attachments: HBASE-15362.patch > > > Compression creates conf using new Configuration() and this leads to it not > respecting the confs set in hbase-site, fixing it is trivial using > HBaseConfiguration.create() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15362) Compression Algorithm does not respect config params from hbase-site
[
https://issues.apache.org/jira/browse/HBASE-15362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172899#comment-15172899
]
Hadoop QA commented on HBASE-15362:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 9s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
39m 30s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 54s
{color} | {color:green} hbase-common in the patch passed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 41s
{color} | {color:green} hbase-common in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 58s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-29 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12790583/HBASE-15362.patch |
| JIRA Issue | HBASE-15362 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 7ac56c37bc52 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3
