[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681356#comment-14681356
]
Hudson commented on HBASE-5878:
---
FAILURE: Integrated in HBase-1.3-IT #82 (See
[https://builds.apache.org/job/HBase-1.3-IT/82/])
HBASE-5878 Use getVisibleLength public api from HdfsDataInputStream from
Hadoop-2. (apurtell: rev 0862abd6599a6936fb8079f4c70afc660175ba11)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0
Attachments: HBASE-5878-branch-1.0.patch, HBASE-5878-v2.patch,
HBASE-5878-v3.patch, HBASE-5878-v4.patch, HBASE-5878-v5-0.98.patch,
HBASE-5878-v5.patch, HBASE-5878-v5.patch, HBASE-5878-v6-0.98.patch,
HBASE-5878-v6.patch, HBASE-5878-v7-0.98.patch, HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681696#comment-14681696
]
Hudson commented on HBASE-5878:
---
FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1027 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1027/])
HBASE-5878 Use getVisibleLength public api from HdfsDataInputStream from Hadoop
2 (apurtell: rev f0e954a4d7e2e96d1fda1d6c5f125db29a552a17)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0
Attachments: HBASE-5878-branch-1.0.patch, HBASE-5878-v2.patch,
HBASE-5878-v3.patch, HBASE-5878-v4.patch, HBASE-5878-v5-0.98.patch,
HBASE-5878-v5.patch, HBASE-5878-v5.patch, HBASE-5878-v6-0.98.patch,
HBASE-5878-v6.patch, HBASE-5878-v7-0.98.patch, HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681980#comment-14681980
]
Nick Dimiduk commented on HBASE-5878:
-
Thanks [~apurtell] and [~ashish singhi]!
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0
Attachments: HBASE-5878-branch-1.0.patch, HBASE-5878-v2.patch,
HBASE-5878-v3.patch, HBASE-5878-v4.patch, HBASE-5878-v5-0.98.patch,
HBASE-5878-v5.patch, HBASE-5878-v5.patch, HBASE-5878-v6-0.98.patch,
HBASE-5878-v6.patch, HBASE-5878-v7-0.98.patch, HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681267#comment-14681267
]
Hudson commented on HBASE-5878:
---
FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1026 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1026/])
HBASE-5878 Use getVisibleLength public api from HdfsDataInputStream from
Hadoop-2. (apurtell: rev b69569f512068d795199310ce662ab381bb6b6b7)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
Revert HBASE-5878 Use getVisibleLength public api from HdfsDataInputStream
from Hadoop-2. (apurtell: rev fabfb423f9cf48ddd52e9583ca6664f42349)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0
Attachments: HBASE-5878-branch-1.0.patch, HBASE-5878-v2.patch,
HBASE-5878-v3.patch, HBASE-5878-v4.patch, HBASE-5878-v5-0.98.patch,
HBASE-5878-v5.patch, HBASE-5878-v5.patch, HBASE-5878-v6-0.98.patch,
HBASE-5878-v6.patch, HBASE-5878-v7-0.98.patch, HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680307#comment-14680307
]
Hadoop QA commented on HBASE-5878:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12749586/HBASE-5878-v7-0.98.patch
against 0.98 branch at commit c7065c4c40e94bcce2035b8ea9813cfc6124a7e0.
ATTACHMENT ID: 12749586
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
22 warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn post-site goal
to fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.wal.TestHLog
{color:red}-1 core zombie tests{color}. There are 4 zombie test(s):
at
org.apache.hadoop.hbase.coprocessor.TestClassLoading.testClassLoadingFromLibDirInJar(TestClassLoading.java:374)
at
org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface.testPreWALRestoreSkip(TestRegionObserverInterface.java:717)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15029//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15029//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15029//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15029//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15029//console
This message is automatically generated.
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.3.0
Attachments: HBASE-5878-branch-1.0.patch, HBASE-5878-v2.patch,
HBASE-5878-v3.patch, HBASE-5878-v4.patch, HBASE-5878-v5-0.98.patch,
HBASE-5878-v5.patch, HBASE-5878-v5.patch, HBASE-5878-v6-0.98.patch,
HBASE-5878-v6.patch, HBASE-5878-v7-0.98.patch, HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength =
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681196#comment-14681196
]
Hudson commented on HBASE-5878:
---
FAILURE: Integrated in HBase-TRUNK #6712 (See
[https://builds.apache.org/job/HBase-TRUNK/6712/])
HBASE-5878 Use getVisibleLength public api from HdfsDataInputStream from
Hadoop-2. (apurtell: rev 6e8cdec242b6c40c09601982bad0a79a569e66c4)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0
Attachments: HBASE-5878-branch-1.0.patch, HBASE-5878-v2.patch,
HBASE-5878-v3.patch, HBASE-5878-v4.patch, HBASE-5878-v5-0.98.patch,
HBASE-5878-v5.patch, HBASE-5878-v5.patch, HBASE-5878-v6-0.98.patch,
HBASE-5878-v6.patch, HBASE-5878-v7-0.98.patch, HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681201#comment-14681201
]
Hudson commented on HBASE-5878:
---
FAILURE: Integrated in HBase-1.2 #99 (See
[https://builds.apache.org/job/HBase-1.2/99/])
HBASE-5878 Use getVisibleLength public api from HdfsDataInputStream from
Hadoop-2. (apurtell: rev 7f33e6330a37b0401c2f9143ddbea67361217453)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0
Attachments: HBASE-5878-branch-1.0.patch, HBASE-5878-v2.patch,
HBASE-5878-v3.patch, HBASE-5878-v4.patch, HBASE-5878-v5-0.98.patch,
HBASE-5878-v5.patch, HBASE-5878-v5.patch, HBASE-5878-v6-0.98.patch,
HBASE-5878-v6.patch, HBASE-5878-v7-0.98.patch, HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681115#comment-14681115
]
Hudson commented on HBASE-5878:
---
FAILURE: Integrated in HBase-1.3 #99 (See
[https://builds.apache.org/job/HBase-1.3/99/])
HBASE-5878 Use getVisibleLength public api from HdfsDataInputStream from
Hadoop-2. (apurtell: rev 0862abd6599a6936fb8079f4c70afc660175ba11)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0
Attachments: HBASE-5878-branch-1.0.patch, HBASE-5878-v2.patch,
HBASE-5878-v3.patch, HBASE-5878-v4.patch, HBASE-5878-v5-0.98.patch,
HBASE-5878-v5.patch, HBASE-5878-v5.patch, HBASE-5878-v6-0.98.patch,
HBASE-5878-v6.patch, HBASE-5878-v7-0.98.patch, HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681135#comment-14681135
]
Hudson commented on HBASE-5878:
---
FAILURE: Integrated in HBase-1.1 #605 (See
[https://builds.apache.org/job/HBase-1.1/605/])
HBASE-5878 Use getVisibleLength public api from HdfsDataInputStream from Hadoop
2 (apurtell: rev 22b96831584d44a63c71c5366f372622d13fd9b4)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0
Attachments: HBASE-5878-branch-1.0.patch, HBASE-5878-v2.patch,
HBASE-5878-v3.patch, HBASE-5878-v4.patch, HBASE-5878-v5-0.98.patch,
HBASE-5878-v5.patch, HBASE-5878-v5.patch, HBASE-5878-v6-0.98.patch,
HBASE-5878-v6.patch, HBASE-5878-v7-0.98.patch, HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681004#comment-14681004
]
Andrew Purtell commented on HBASE-5878:
---
I reverted the 0.98 patch from 0.98 and applied the branch-1.0 patch there
instead, so the result here is the new changes from branch-1.2 and up, only the
one line change to rethrow an exception if we fail to read the file length from
branch-1.1 back. I think that's a good outcome.
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0
Attachments: HBASE-5878-branch-1.0.patch, HBASE-5878-v2.patch,
HBASE-5878-v3.patch, HBASE-5878-v4.patch, HBASE-5878-v5-0.98.patch,
HBASE-5878-v5.patch, HBASE-5878-v5.patch, HBASE-5878-v6-0.98.patch,
HBASE-5878-v6.patch, HBASE-5878-v7-0.98.patch, HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681100#comment-14681100
]
Hudson commented on HBASE-5878:
---
SUCCESS: Integrated in HBase-1.2-IT #82 (See
[https://builds.apache.org/job/HBase-1.2-IT/82/])
HBASE-5878 Use getVisibleLength public api from HdfsDataInputStream from
Hadoop-2. (apurtell: rev 7f33e6330a37b0401c2f9143ddbea67361217453)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0
Attachments: HBASE-5878-branch-1.0.patch, HBASE-5878-v2.patch,
HBASE-5878-v3.patch, HBASE-5878-v4.patch, HBASE-5878-v5-0.98.patch,
HBASE-5878-v5.patch, HBASE-5878-v5.patch, HBASE-5878-v6-0.98.patch,
HBASE-5878-v6.patch, HBASE-5878-v7-0.98.patch, HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680998#comment-14680998
]
Andrew Purtell commented on HBASE-5878:
---
Missed TestHLog because it's outside of o.a.h.h.regionserver.wal.* . I'm going
to revert the 0.98 commit. Let's just forget about it there.
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.0.2, 1.2.0, 1.1.2, 1.3.0
Attachments: HBASE-5878-branch-1.0.patch, HBASE-5878-v2.patch,
HBASE-5878-v3.patch, HBASE-5878-v4.patch, HBASE-5878-v5-0.98.patch,
HBASE-5878-v5.patch, HBASE-5878-v5.patch, HBASE-5878-v6-0.98.patch,
HBASE-5878-v6.patch, HBASE-5878-v7-0.98.patch, HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680082#comment-14680082
]
Ashish Singhi commented on HBASE-5878:
--
[~andrew.purt...@gmail.com], I have attached 0.98 version branch patch(v7) to
support both hadoop 1 and hadoop 2.
Please check if this looks ok.
Thanks.
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.3.0
Attachments: HBASE-5878-branch-1.0.patch, HBASE-5878-v2.patch,
HBASE-5878-v3.patch, HBASE-5878-v4.patch, HBASE-5878-v5-0.98.patch,
HBASE-5878-v5.patch, HBASE-5878-v5.patch, HBASE-5878-v6-0.98.patch,
HBASE-5878-v6.patch, HBASE-5878-v7-0.98.patch, HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14662637#comment-14662637
]
Andrew Purtell commented on HBASE-5878:
---
We can't take the 0.98 patch as is because it doesn't compile against Hadoop 1.
You'll need to keep the current code and use reflection to detect and use the
new getVisibleLength API [~ashish singhi].
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.3.0
Attachments: HBASE-5878-v2.patch, HBASE-5878-v3.patch,
HBASE-5878-v4.patch, HBASE-5878-v5-0.98.patch, HBASE-5878-v5.patch,
HBASE-5878-v5.patch, HBASE-5878-v6-0.98.patch, HBASE-5878-v6.patch,
HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14662748#comment-14662748
]
Hadoop QA commented on HBASE-5878:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12749369/HBASE-5878-branch-1.0.patch
against branch-1.0 branch at commit ea8833fd63c9ef35292766f646b1e507577e46f5.
ATTACHMENT ID: 12749369
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.util.TestProcessBasedCluster
org.apache.hadoop.hbase.mapreduce.TestImportExport
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15006//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15006//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15006//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15006//console
This message is automatically generated.
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.3.0
Attachments: HBASE-5878-branch-1.0.patch, HBASE-5878-v2.patch,
HBASE-5878-v3.patch, HBASE-5878-v4.patch, HBASE-5878-v5-0.98.patch,
HBASE-5878-v5.patch, HBASE-5878-v5.patch, HBASE-5878-v6-0.98.patch,
HBASE-5878-v6.patch, HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660755#comment-14660755
]
Hadoop QA commented on HBASE-5878:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12749093/HBASE-5878-v5-0.98.patch
against 0.98 branch at commit 7a9e10dc11877420c53245c403897d746bebc077.
ATTACHMENT ID: 12749093
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
22 warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
3879 checkstyle errors (more than the master's current 3877 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn post-site goal
to fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.rest.TestTableResource.testTableListXML(TestTableResource.java:207)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14992//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14992//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14992//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14992//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14992//console
This message is automatically generated.
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.3.0
Attachments: HBASE-5878-v2.patch, HBASE-5878-v3.patch,
HBASE-5878-v4.patch, HBASE-5878-v5-0.98.patch, HBASE-5878-v5.patch,
HBASE-5878-v5.patch, HBASE-5878-v6-0.98.patch, HBASE-5878-v6.patch,
HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660301#comment-14660301
]
Andrew Purtell commented on HBASE-5878:
---
That's fine by me [~ndimiduk].
What about taking in a modified patch that just rethrows the exception?
{code}
- } catch(Exception e) {
- SequenceFileLogReader.LOG.warn(
-Error while trying to get accurate file length. +
-Truncation / data loss may occur if RegionServers die., e);
{code}
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.3.0
Attachments: HBASE-5878-v2.patch, HBASE-5878-v3.patch,
HBASE-5878-v4.patch, HBASE-5878-v5-0.98.patch, HBASE-5878-v5.patch,
HBASE-5878-v5.patch, HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660582#comment-14660582
]
Andrew Purtell commented on HBASE-5878:
---
bq. I can wrap the exception in a new IOE and throw that, is this ok for all
the fix versions
That could work. What do you think [~ndimiduk] ? Continuing after likely data
loss seems like a bug.
I missed this before:
bq. Patch applies mostly cleanly to 0.98, attaching here. You want this back
ported Andrew Purtell?
Sure, we can take this change there
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.3.0
Attachments: HBASE-5878-v2.patch, HBASE-5878-v3.patch,
HBASE-5878-v4.patch, HBASE-5878-v5-0.98.patch, HBASE-5878-v5.patch,
HBASE-5878-v5.patch, HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660422#comment-14660422
]
Ashish Singhi commented on HBASE-5878:
--
[~apurtell], we cannot throw the exception as is, as the method is overridden
from
[FSDataInputStream|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java]
which throws IOE.
I can wrap the exception in a new IOE and throw that, is this ok for all the
fix versions ? Just want confirm it before I attach a new patch.
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.3.0
Attachments: HBASE-5878-v2.patch, HBASE-5878-v3.patch,
HBASE-5878-v4.patch, HBASE-5878-v5-0.98.patch, HBASE-5878-v5.patch,
HBASE-5878-v5.patch, HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660634#comment-14660634
]
Nick Dimiduk commented on HBASE-5878:
-
bq. Continuing after likely data loss seems like a bug
Agreed. Better safe than sorry, better to throw the exception.
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.3.0
Attachments: HBASE-5878-v2.patch, HBASE-5878-v3.patch,
HBASE-5878-v4.patch, HBASE-5878-v5-0.98.patch, HBASE-5878-v5.patch,
HBASE-5878-v5.patch, HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660252#comment-14660252
]
Nick Dimiduk commented on HBASE-5878:
-
Thanks [~ashish singhi]. Patch applies, builds cleanly on branch-1 as well. I
will commit this tomorrow unless there are specific objections.
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.1.2, 1.3.0, 1.2.1, 1.0.3
Attachments: HBASE-5878-v2.patch, HBASE-5878-v3.patch,
HBASE-5878-v4.patch, HBASE-5878-v5.patch, HBASE-5878-v5.patch,
HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14658488#comment-14658488
]
Nick Dimiduk commented on HBASE-5878:
-
What's the dependency on HDFS-3184 -- is that really a dependency before we
commit this?
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.1.2, 1.3.0, 1.2.1, 1.0.3
Attachments: HBASE-5878-v2.patch, HBASE-5878-v3.patch,
HBASE-5878-v4.patch, HBASE-5878-v5.patch, HBASE-5878-v5.patch,
HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14658525#comment-14658525
]
Ashish Singhi commented on HBASE-5878:
--
I believe the dependency we had was to get a publicly exposed api. Which was
done as part of one of the sub task from HDFS-3184 and we are using that in
this patch.
(sent from mobile excuse if any typos)
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.1.2, 1.3.0, 1.2.1, 1.0.3
Attachments: HBASE-5878-v2.patch, HBASE-5878-v3.patch,
HBASE-5878-v4.patch, HBASE-5878-v5.patch, HBASE-5878-v5.patch,
HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14658921#comment-14658921
]
Nick Dimiduk commented on HBASE-5878:
-
It would be good to update the dependency to the specific JIRA so as to avoid
confusion for lookers-on.
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.1.2, 1.3.0, 1.2.1, 1.0.3
Attachments: HBASE-5878-v2.patch, HBASE-5878-v3.patch,
HBASE-5878-v4.patch, HBASE-5878-v5.patch, HBASE-5878-v5.patch,
HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14659496#comment-14659496
]
Ashish Singhi commented on HBASE-5878:
--
Done.
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.1.2, 1.3.0, 1.2.1, 1.0.3
Attachments: HBASE-5878-v2.patch, HBASE-5878-v3.patch,
HBASE-5878-v4.patch, HBASE-5878-v5.patch, HBASE-5878-v5.patch,
HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14654844#comment-14654844
]
Hadoop QA commented on HBASE-5878:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12748763/HBASE-5878-v5.patch
against master branch at commit 5f6632f80159f283125a7a826d5f8ef76dbe1caa.
ATTACHMENT ID: 12748763
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14978//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14978//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14978//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14978//console
This message is automatically generated.
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.1.2, 1.3.0, 1.2.1, 1.0.3
Attachments: HBASE-5878-v2.patch, HBASE-5878-v3.patch,
HBASE-5878-v4.patch, HBASE-5878-v5.patch, HBASE-5878-v5.patch,
HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14654642#comment-14654642
]
Nick Dimiduk commented on HBASE-5878:
-
Buildbot flakiness aside, are reviewers good with this patch? Ping [~eclark],
[~stack].
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.1.2, 1.3.0, 1.2.1, 1.0.3
Attachments: HBASE-5878-v2.patch, HBASE-5878-v3.patch,
HBASE-5878-v4.patch, HBASE-5878-v5.patch, HBASE-5878-v5.patch,
HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14613599#comment-14613599
]
Hadoop QA commented on HBASE-5878:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12743500/HBASE-5878-v5.patch
against master branch at commit e640f1e76af8f32015f475629610da127897f01e.
ATTACHMENT ID: 12743500
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14652//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14652//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14652//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14652//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14652//console
This message is automatically generated.
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.1.2, 1.3.0, 1.2.1, 1.0.3
Attachments: HBASE-5878-v2.patch, HBASE-5878-v3.patch,
HBASE-5878-v4.patch, HBASE-5878-v5.patch, HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614013#comment-14614013
]
Ashish Singhi commented on HBASE-5878:
--
-1's not related to patch.
Build was successful
{noformat}
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] HBase . SUCCESS [ 1.480 s]
[INFO] HBase - Checkstyle SUCCESS [ 0.052 s]
[INFO] HBase - Annotations ... SUCCESS [ 0.076 s]
[INFO] HBase - Protocol .. SUCCESS [ 0.121 s]
[INFO] HBase - Common SUCCESS [ 0.219 s]
[INFO] HBase - Procedure . SUCCESS [ 0.072 s]
[INFO] HBase - Client SUCCESS [ 0.296 s]
[INFO] HBase - Hadoop Compatibility .. SUCCESS [ 0.075 s]
[INFO] HBase - Hadoop Two Compatibility .. SUCCESS [ 0.126 s]
[INFO] HBase - Prefix Tree ... SUCCESS [ 0.097 s]
[INFO] HBase - Server SUCCESS [ 0.964 s]
[INFO] HBase - Testing Util .. SUCCESS [ 0.037 s]
[INFO] HBase - Thrift SUCCESS [ 0.084 s]
[INFO] HBase - Shell . SUCCESS [ 0.099 s]
[INFO] HBase - Integration Tests . SUCCESS [ 0.097 s]
[INFO] HBase - Examples .. SUCCESS [ 0.061 s]
[INFO] HBase - Rest .. SUCCESS [ 0.104 s]
[INFO] HBase - Assembly .. SUCCESS [ 0.042 s]
[INFO] HBase - Shaded SUCCESS [ 0.036 s]
[INFO] HBase - Shaded - Client ... SUCCESS [ 0.035 s]
[INFO] HBase - Shaded - Server ... SUCCESS [ 0.035 s]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 6.747 s
[INFO] Finished at: 2015-07-04T08:04:56+00:00
[INFO] Final Memory: 36M/913M
[INFO]
{noformat}
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.1.2, 1.3.0, 1.2.1, 1.0.3
Attachments: HBASE-5878-v2.patch, HBASE-5878-v3.patch,
HBASE-5878-v4.patch, HBASE-5878-v5.patch, HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14613090#comment-14613090
]
Ashish Singhi commented on HBASE-5878:
--
Attached v5 patch same as v4 patch.
Please review.
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.1.2, 1.3.0, 1.2.1, 1.0.3
Attachments: HBASE-5878-v2.patch, HBASE-5878-v3.patch,
HBASE-5878-v4.patch, HBASE-5878-v5.patch, HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14276606#comment-14276606
]
Hadoop QA commented on HBASE-5878:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12692160/HBASE-5878-v4.patch
against master branch at commit 9b7f36b8cf521bcc01ac6476349a9d2f34be8bb3.
ATTACHMENT ID: 12692160
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.curator.test.TestingZooKeeperMain.runFromConfig(TestingZooKeeperMain.java:73)
at
org.apache.curator.test.TestingZooKeeperServer$1.run(TestingZooKeeperServer.java:134)
at org.apache.oozie.test.MiniHCatServer$1.run(MiniHCatServer.java:137)
at
org.apache.oozie.test.XTestCase$MiniClusterShutdownMonitor.run(XTestCase.java:1071)
at org.apache.oozie.test.XTestCase.waitFor(XTestCase.java:692)
at org.apache.oozie.test.XTestCase.sleep(XTestCase.java:710)
at
org.apache.oozie.service.TestZKXLogStreamingService.testDisableLogOverWS(TestZKXLogStreamingService.java:88)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12458//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12458//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12458//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12458//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12458//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12458//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12458//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12458//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12458//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12458//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12458//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12458//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12458//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12458//console
This message is automatically generated.
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 1.0.0, 2.0.0
Attachments: HBASE-5878-v2.patch, HBASE-5878-v3.patch,
HBASE-5878-v4.patch,
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14275720#comment-14275720
]
Elliott Clark commented on HBASE-5878:
--
Will it ever enter the top of the if statement?
Here DFS is uppercase.
{code}.endsWith(DFSInputStream){code}
However here it's lowercase dfs.
{code}hdfsDataInputStream = (HdfsDataInputStream) this.getWrappedStream();{code}
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 1.0.0, 2.0.0
Attachments: HBASE-5878-v2.patch, HBASE-5878-v3.patch,
HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14275525#comment-14275525
]
Ashish Singhi commented on HBASE-5878:
--
@Stack can you please review v3 patch.
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 1.0.0, 2.0.0
Attachments: HBASE-5878-v2.patch, HBASE-5878-v3.patch,
HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270694#comment-14270694
]
Hadoop QA commented on HBASE-5878:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12691048/HBASE-5878-v3.patch
against master branch at commit 4bfd35aabc83b007871fe9a8537cd5ac9cd84500.
ATTACHMENT ID: 12691048
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12382//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12382//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12382//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12382//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12382//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12382//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12382//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12382//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12382//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12382//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12382//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12382//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12382//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12382//console
This message is automatically generated.
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 1.0.0, 2.0.0
Attachments: HBASE-5878-v2.patch, HBASE-5878-v3.patch,
HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn =
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270565#comment-14270565
]
Hadoop QA commented on HBASE-5878:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12691031/HBASE-5878-v2.patch
against master branch at commit 4bfd35aabc83b007871fe9a8537cd5ac9cd84500.
ATTACHMENT ID: 12691031
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12378//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12378//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12378//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12378//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12378//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12378//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12378//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12378//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12378//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12378//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12378//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12378//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12378//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12378//console
This message is automatically generated.
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 1.0.0, 2.0.0
Attachments: HBASE-5878-v2.patch, HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268958#comment-14268958
]
Hadoop QA commented on HBASE-5878:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12690726/HBASE-5878.patch
against master branch at commit 645fbd7d87450b31b67b1e535cdb9c1cf50ffd16.
ATTACHMENT ID: 12690726
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12358//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12358//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12358//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12358//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12358//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12358//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12358//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12358//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12358//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12358//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12358//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12358//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12358//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12358//console
This message is automatically generated.
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 1.0.0, 2.0.0
Attachments: HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270052#comment-14270052
]
stack commented on HBASE-5878:
--
Is getFileLength same as getVisibleLength? What is minimum hadoop that has this
API? Thanks.
How about just letting the exception out:
120 SequenceFileLogReader.LOG.warn(Error while trying to get
accurate file length.
121 + Truncation / data loss may occur if RegionServers
die., e);
Since it is dataloss (as suggested above by [~umamaheswararao]?)
Otherwise patch looks good.
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 1.0.0, 2.0.0
Attachments: HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14267705#comment-14267705
]
Cosmin Lehene commented on HBASE-5878:
--
[~umamaheswararao] did you create a patch for this?
Looking at HDFS-3184 umbrella it seems that it's mostly done and given we're
removing support for older versions of hadoop I guess we can (and should)
refactor this.
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Fix For: 1.0.0
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263361#comment-13263361
]
Jieshan Bean commented on HBASE-5878:
-
bq.We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
+1 on this idea.
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
