[jira] [Commented] (HBASE-15122) Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
[
https://issues.apache.org/jira/browse/HBASE-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147033#comment-15147033
]
Hudson commented on HBASE-15122:
SUCCESS: Integrated in HBase-1.2-IT #437 (See
[https://builds.apache.org/job/HBase-1.2-IT/437/])
HBASE-15122 Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER
(chenheng: rev 3733223e409c5addfcbb914c202d8131f5ee95e8)
* hbase-server/src/main/resources/ESAPI.properties
* pom.xml
* hbase-server/src/test/resources/ESAPI.properties
*
hbase-server/src/main/java/org/apache/hadoop/hbase/http/jmx/JMXJsonServlet.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/http/jmx/TestJMXJsonServlet.java
* hbase-server/pom.xml
* hbase-resource-bundle/src/main/resources/supplemental-models.xml
> Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
> ---
>
> Key: HBASE-15122
> URL: https://issues.apache.org/jira/browse/HBASE-15122
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: Samir Ahmic
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4
>
> Attachments: HBASE-15122-v0-master, HBASE-15122.patch,
> HBASE-15122_v1.patch, HBASE-15122_v2.patch, HBASE-15122_v3.patch
>
>
> In our JMXJsonServlet we are doing this:
> jsonpcb = request.getParameter(CALLBACK_PARAM);
> if (jsonpcb != null) {
> response.setContentType("application/javascript; charset=utf8");
> writer.write(jsonpcb + "(");
> ...
> Findbugs complains rightly. There are other instances in our servlets and
> then there are the pages generated by jamon excluded from findbugs checking
> (and findbugs volunteers that it is dumb in this regard finding only the most
> egregious of violations).
> We have no sanitizing tooling in hbase that I know of (correct me if I am
> wrong). I started to pull on this thread and it runs deep. Our Jamon
> templating (last updated in 2013 and before that, in 2011) engine doesn't
> seem to have sanitizing means either and there seems to be outstanding XSS
> complaint against jamon that goes unaddressed.
> Could pull in something like
> https://www.owasp.org/index.php/OWASP_Java_Encoder_Project and run all
> emissions via it or get a templating engine that has sanitizing built in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15267) Read requests count metric is increased when Get operation does not return any record
[ https://issues.apache.org/jira/browse/HBASE-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147004#comment-15147004 ] Anoop Sam John commented on HBASE-15267: That is being tracked by 'filteredReadRequestsCount' right? Seems that is correctly incremented. Ya 'readRequestsCount' which has to track how many reads happen should get incremented irrespective of read return some cells or not.. Not getting what you are trying to fix. Sorry > Read requests count metric is increased when Get operation does not return > any record > - > > Key: HBASE-15267 > URL: https://issues.apache.org/jira/browse/HBASE-15267 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15267-v1.patch, HBASE-15267.patch > > > Read requests count is increased when Get operation returns no record by > filtering out. In such cases, Get for deleted record, Get for TTL expired > record and Get for filtering out. > So I fixed the bug and added some more test cases for read metrics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15219) Canary tool does not return non-zero exit code when one of regions is in stuck state
[
https://issues.apache.org/jira/browse/HBASE-15219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147001#comment-15147001
]
Maddineni Sukumar commented on HBASE-15219:
---
Tested latest v9 patch and its giving return codes properly. So far every thing
looks good. Will be doing some long running tests in next couple of days and
let you know if there are any findings.
Thanks [~ted_yu]
> Canary tool does not return non-zero exit code when one of regions is in
> stuck state
> -
>
> Key: HBASE-15219
> URL: https://issues.apache.org/jira/browse/HBASE-15219
> Project: HBase
> Issue Type: Improvement
> Components: canary
>Affects Versions: 0.98.16
>Reporter: Vishal Khandelwal
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 0.98.18
>
> Attachments: HBASE-15219-branch-1.2.v8.patch, HBASE-15219.v1.patch,
> HBASE-15219.v3.patch, HBASE-15219.v4.patch, HBASE-15219.v5.patch,
> HBASE-15219.v7.patch, HBASE-15219.v8.patch, HBASE-15219.v9.patch
>
>
> {code}
> 2016-02-05 12:24:18,571 ERROR [pool-2-thread-7] tool.Canary - read from
> region
> CAN_1,\x08\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1454667477865.00e77d07b8defe10704417fb99aa0418.
> column family 0 failed
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=2, exceptions:
> Fri Feb 05 12:24:15 GMT 2016,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@54c9fea0,
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> CAN_1,\x08\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1454667477865.00e77d07b8defe10704417fb99aa0418.
> is not online on isthbase02-dnds1-3-crd.eng.sfdc.net,60020,1454669984738
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2852)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4468)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2984)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31186)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2149)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:104)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
>
> -bash-4.1$ echo $?
> 0
> {code}
> Below code prints the error but it does sets/returns the exit code. Due to
> this tool can't be integrated with nagios or other alerting.
> Ideally it should return error for failures. as pre the documentation:
>
> This tool will return non zero error codes to user for collaborating with
> other monitoring tools, such as Nagios. The error code definitions are:
> private static final int USAGE_EXIT_CODE = 1;
> private static final int INIT_ERROR_EXIT_CODE = 2;
> private static final int TIMEOUT_ERROR_EXIT_CODE = 3;
> private static final int ERROR_EXIT_CODE = 4;
>
> {code}
> org.apache.hadoop.hbase.tool.Canary.RegionTask
> public Void read() {
>
> try {
> table = connection.getTable(region.getTable());
> tableDesc = table.getTableDescriptor();
> } catch (IOException e) {
> LOG.debug("sniffRegion failed", e);
> sink.publishReadFailure(region, e);
>...
> return null;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15122) Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
[
https://issues.apache.org/jira/browse/HBASE-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146984#comment-15146984
]
Hudson commented on HBASE-15122:
SUCCESS: Integrated in HBase-1.3-IT #502 (See
[https://builds.apache.org/job/HBase-1.3-IT/502/])
HBASE-15122 Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER
(chenheng: rev 29ce46a67fc61f42d06d8d9fed507f65c5d620c4)
* hbase-server/src/test/resources/ESAPI.properties
*
hbase-server/src/test/java/org/apache/hadoop/hbase/http/jmx/TestJMXJsonServlet.java
* pom.xml
* hbase-server/pom.xml
* hbase-resource-bundle/src/main/resources/supplemental-models.xml
* hbase-server/src/main/resources/ESAPI.properties
*
hbase-server/src/main/java/org/apache/hadoop/hbase/http/jmx/JMXJsonServlet.java
> Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
> ---
>
> Key: HBASE-15122
> URL: https://issues.apache.org/jira/browse/HBASE-15122
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: Samir Ahmic
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4
>
> Attachments: HBASE-15122-v0-master, HBASE-15122.patch,
> HBASE-15122_v1.patch, HBASE-15122_v2.patch, HBASE-15122_v3.patch
>
>
> In our JMXJsonServlet we are doing this:
> jsonpcb = request.getParameter(CALLBACK_PARAM);
> if (jsonpcb != null) {
> response.setContentType("application/javascript; charset=utf8");
> writer.write(jsonpcb + "(");
> ...
> Findbugs complains rightly. There are other instances in our servlets and
> then there are the pages generated by jamon excluded from findbugs checking
> (and findbugs volunteers that it is dumb in this regard finding only the most
> egregious of violations).
> We have no sanitizing tooling in hbase that I know of (correct me if I am
> wrong). I started to pull on this thread and it runs deep. Our Jamon
> templating (last updated in 2013 and before that, in 2011) engine doesn't
> seem to have sanitizing means either and there seems to be outstanding XSS
> complaint against jamon that goes unaddressed.
> Could pull in something like
> https://www.owasp.org/index.php/OWASP_Java_Encoder_Project and run all
> emissions via it or get a templating engine that has sanitizing built in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15169) Backport HBASE-14362 'TestWALProcedureStoreOnHDFS is super duper flaky' to branch-1.1
[ https://issues.apache.org/jira/browse/HBASE-15169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-15169: -- Attachment: HBASE-15169-branch-1.1.patch > Backport HBASE-14362 'TestWALProcedureStoreOnHDFS is super duper flaky' to > branch-1.1 > - > > Key: HBASE-15169 > URL: https://issues.apache.org/jira/browse/HBASE-15169 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Nick Dimiduk >Assignee: Heng Chen >Priority: Critical > Fix For: 1.1.4 > > Attachments: HBASE-15169-branch-1.1.patch, > HBASE-15169-branch-1.1.patch, HBASE-15169-branch-1.1.patch, > HBASE-15169-branch-1.1.patch, HBASE-15169-branch-1.1.patch > > > This one fails consistently for me and there's a fix hanging out upstream. > Let's bring it back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15180) Reduce garbage created while reading Cells from Codec Decoder
[ https://issues.apache.org/jira/browse/HBASE-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146976#comment-15146976 ] Anoop Sam John commented on HBASE-15180: Ya my mistake in selecting Eclipse import suggestion .. Will correct. Thanks for the good eye. > Reduce garbage created while reading Cells from Codec Decoder > - > > Key: HBASE-15180 > URL: https://issues.apache.org/jira/browse/HBASE-15180 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 0.98.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-15180.patch, HBASE-15180_V2.patch, > HBASE-15180_V4.patch > > > In KeyValueDecoder#parseCell (Default Codec decoder) we use > KeyValueUtil#iscreate to read cells from the InputStream. Here we 1st create > a byte[] of length 4 and read the cell length and then an array of Cell's > length and read in cell bytes into it and create a KV. > Actually in server we read the reqs into a byte[] and CellScanner is created > on top of a ByteArrayInputStream on top of this. By default in write path, we > have MSLAB usage ON. So while adding Cells to memstore, we will copy the Cell > bytes to MSLAB memory chunks (default 2 MB size) and recreate Cells over that > bytes. So there is no issue if we create Cells over the RPC read byte[] > directly here in Decoder. No need for 2 byte[] creation and copy for every > Cell in request. > My plan is to make a Cell aware ByteArrayInputStream which can read Cells > directly from it. > Same Codec path is used in client side also. There better we can avoid this > direct Cell create and continue to do the copy to smaller byte[]s path. Plan > to introduce some thing like a CodecContext associated with every Codec > instance which can say the server/client context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146977#comment-15146977 ] Ashish Singhi commented on HBASE-9393: -- bq. First of all, please make sure discussions get reflected in a public place (like this jira or dev@). Not just the decision, but the reasoning is important so that others can chime in. All the discussions, reviews and comments are put up in the jira. The offline discussion was only about handling these in HBase versions which by default have Hadoop jars prior to 2.7.1 version and this point was also noted down in the jira. The patch was implemented based on one of the Colin suggestion(#2) and we have commented the reason also for not going with other suggestion. bq. Please don't catch and then discard Throwable I agree and can handle this once we have a conclusion on this jira. Pardon me for my oversight on CanBuffer interface compatibility check. And I saw that Ted has already raised HADOOP-12805 to handle the interface compatibility. > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v2.patch, HBASE-9393.v3.patch, HBASE-9393.v4.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15122) Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
[
https://issues.apache.org/jira/browse/HBASE-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-15122:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
push to master and branch-1+
> Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
> ---
>
> Key: HBASE-15122
> URL: https://issues.apache.org/jira/browse/HBASE-15122
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: Samir Ahmic
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4
>
> Attachments: HBASE-15122-v0-master, HBASE-15122.patch,
> HBASE-15122_v1.patch, HBASE-15122_v2.patch, HBASE-15122_v3.patch
>
>
> In our JMXJsonServlet we are doing this:
> jsonpcb = request.getParameter(CALLBACK_PARAM);
> if (jsonpcb != null) {
> response.setContentType("application/javascript; charset=utf8");
> writer.write(jsonpcb + "(");
> ...
> Findbugs complains rightly. There are other instances in our servlets and
> then there are the pages generated by jamon excluded from findbugs checking
> (and findbugs volunteers that it is dumb in this regard finding only the most
> egregious of violations).
> We have no sanitizing tooling in hbase that I know of (correct me if I am
> wrong). I started to pull on this thread and it runs deep. Our Jamon
> templating (last updated in 2013 and before that, in 2011) engine doesn't
> seem to have sanitizing means either and there seems to be outstanding XSS
> complaint against jamon that goes unaddressed.
> Could pull in something like
> https://www.owasp.org/index.php/OWASP_Java_Encoder_Project and run all
> emissions via it or get a templating engine that has sanitizing built in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15259) WALEdits under replay will also be replicated
[
https://issues.apache.org/jira/browse/HBASE-15259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146943#comment-15146943
]
Heng Chen commented on HBASE-15259:
---
NO... I just test it with testcase...
> WALEdits under replay will also be replicated
> -
>
> Key: HBASE-15259
> URL: https://issues.apache.org/jira/browse/HBASE-15259
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Attachments: HBASE-15259.patch
>
>
> I need to verify this but seeing the code
> {code}
> try {
> // We are about to append this edit; update the region-scoped
> sequence number. Do it
> // here inside this single appending/writing thread. Events are
> ordered on the ringbuffer
> // so region sequenceids will also be in order.
> regionSequenceId = entry.stampRegionSequenceId();
> // Edits are empty, there is nothing to append. Maybe empty when we
> are looking for a
> // region sequence id only, a region edit/sequence id that is not
> associated with an actual
> // edit. It has to go through all the rigmarole to be sure we have
> the right ordering.
> if (entry.getEdit().isEmpty()) {
> return;
> }
> // Coprocessor hook.
> if (!coprocessorHost.preWALWrite(entry.getHRegionInfo(),
> entry.getKey(),
> entry.getEdit())) {
> if (entry.getEdit().isReplay()) {
> // Set replication scope null so that this won't be replicated
> entry.getKey().setScopes(null);
> }
> }
> if (!listeners.isEmpty()) {
> for (WALActionsListener i: listeners) {
> // TODO: Why does listener take a table description and CPs take
> a regioninfo? Fix.
> i.visitLogEntryBeforeWrite(entry.getHTableDescriptor(),
> entry.getKey(),
> entry.getEdit());
> }
> }
> {code}
> When a WALEdit is in replay we set the Logkey to null. But in the
> visitLogEntryBeforeWrite() we again set the LogKey based on the replication
> scope associated with the cells. So the previous step of setting null does
> not work here?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15259) WALEdits under replay will also be replicated
[
https://issues.apache.org/jira/browse/HBASE-15259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146941#comment-15146941
]
Anoop Sam John commented on HBASE-15259:
You tested it on real cluster right?
> WALEdits under replay will also be replicated
> -
>
> Key: HBASE-15259
> URL: https://issues.apache.org/jira/browse/HBASE-15259
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Attachments: HBASE-15259.patch
>
>
> I need to verify this but seeing the code
> {code}
> try {
> // We are about to append this edit; update the region-scoped
> sequence number. Do it
> // here inside this single appending/writing thread. Events are
> ordered on the ringbuffer
> // so region sequenceids will also be in order.
> regionSequenceId = entry.stampRegionSequenceId();
> // Edits are empty, there is nothing to append. Maybe empty when we
> are looking for a
> // region sequence id only, a region edit/sequence id that is not
> associated with an actual
> // edit. It has to go through all the rigmarole to be sure we have
> the right ordering.
> if (entry.getEdit().isEmpty()) {
> return;
> }
> // Coprocessor hook.
> if (!coprocessorHost.preWALWrite(entry.getHRegionInfo(),
> entry.getKey(),
> entry.getEdit())) {
> if (entry.getEdit().isReplay()) {
> // Set replication scope null so that this won't be replicated
> entry.getKey().setScopes(null);
> }
> }
> if (!listeners.isEmpty()) {
> for (WALActionsListener i: listeners) {
> // TODO: Why does listener take a table description and CPs take
> a regioninfo? Fix.
> i.visitLogEntryBeforeWrite(entry.getHTableDescriptor(),
> entry.getKey(),
> entry.getEdit());
> }
> }
> {code}
> When a WALEdit is in replay we set the Logkey to null. But in the
> visitLogEntryBeforeWrite() we again set the LogKey based on the replication
> scope associated with the cells. So the previous step of setting null does
> not work here?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15122) Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
[
https://issues.apache.org/jira/browse/HBASE-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-15122:
--
Fix Version/s: 1.0.4
1.1.4
1.2.1
1.3.0
2.0.0
> Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
> ---
>
> Key: HBASE-15122
> URL: https://issues.apache.org/jira/browse/HBASE-15122
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: Samir Ahmic
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4
>
> Attachments: HBASE-15122-v0-master, HBASE-15122.patch,
> HBASE-15122_v1.patch, HBASE-15122_v2.patch, HBASE-15122_v3.patch
>
>
> In our JMXJsonServlet we are doing this:
> jsonpcb = request.getParameter(CALLBACK_PARAM);
> if (jsonpcb != null) {
> response.setContentType("application/javascript; charset=utf8");
> writer.write(jsonpcb + "(");
> ...
> Findbugs complains rightly. There are other instances in our servlets and
> then there are the pages generated by jamon excluded from findbugs checking
> (and findbugs volunteers that it is dumb in this regard finding only the most
> egregious of violations).
> We have no sanitizing tooling in hbase that I know of (correct me if I am
> wrong). I started to pull on this thread and it runs deep. Our Jamon
> templating (last updated in 2013 and before that, in 2011) engine doesn't
> seem to have sanitizing means either and there seems to be outstanding XSS
> complaint against jamon that goes unaddressed.
> Could pull in something like
> https://www.owasp.org/index.php/OWASP_Java_Encoder_Project and run all
> emissions via it or get a templating engine that has sanitizing built in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15122) Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
[
https://issues.apache.org/jira/browse/HBASE-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146926#comment-15146926
]
stack commented on HBASE-15122:
---
+1
Nice work [~asamir]
Suggest it go back on branch-1 too.
I can see us making use of this lib in a good few places going forward.
> Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
> ---
>
> Key: HBASE-15122
> URL: https://issues.apache.org/jira/browse/HBASE-15122
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: Samir Ahmic
>Priority: Critical
> Attachments: HBASE-15122-v0-master, HBASE-15122.patch,
> HBASE-15122_v1.patch, HBASE-15122_v2.patch, HBASE-15122_v3.patch
>
>
> In our JMXJsonServlet we are doing this:
> jsonpcb = request.getParameter(CALLBACK_PARAM);
> if (jsonpcb != null) {
> response.setContentType("application/javascript; charset=utf8");
> writer.write(jsonpcb + "(");
> ...
> Findbugs complains rightly. There are other instances in our servlets and
> then there are the pages generated by jamon excluded from findbugs checking
> (and findbugs volunteers that it is dumb in this regard finding only the most
> egregious of violations).
> We have no sanitizing tooling in hbase that I know of (correct me if I am
> wrong). I started to pull on this thread and it runs deep. Our Jamon
> templating (last updated in 2013 and before that, in 2011) engine doesn't
> seem to have sanitizing means either and there seems to be outstanding XSS
> complaint against jamon that goes unaddressed.
> Could pull in something like
> https://www.owasp.org/index.php/OWASP_Java_Encoder_Project and run all
> emissions via it or get a templating engine that has sanitizing built in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15263) TestIPv6NIOServerSocketChannel.testServerSocketFromLocalhostResolution can hang indefinetly
[
https://issues.apache.org/jira/browse/HBASE-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146923#comment-15146923
]
stack commented on HBASE-15263:
---
[~chenheng] I should figure where the loop is...
> TestIPv6NIOServerSocketChannel.testServerSocketFromLocalhostResolution can
> hang indefinetly
> ---
>
> Key: HBASE-15263
> URL: https://issues.apache.org/jira/browse/HBASE-15263
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>
> Should have a timeout on this test at least. Stuck here:
> {code}
> "main" #1 prio=5 os_prio=0 tid=0x7f3f9c008000 nid=0xca97 runnable
> [0x7f3fa2e8c000]
>java.lang.Thread.State: RUNNABLE
> at
> org.apache.hadoop.hbase.HBaseTestingUtility$PortAllocator.randomFreePort(HBaseTestingUtility.java:3500)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.randomFreePort(HBaseTestingUtility.java:3450)
> at
> org.apache.hadoop.hbase.TestIPv6NIOServerSocketChannel.bindServerSocket(TestIPv6NIOServerSocketChannel.java:57)
> at
> org.apache.hadoop.hbase.TestIPv6NIOServerSocketChannel.testServerSocketFromLocalhostResolution(TestIPv6NIOServerSocketChannel.java:151)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:108)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:78)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:54)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:144)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14919) Infrastructure refactoring for In-Memory Flush
[ https://issues.apache.org/jira/browse/HBASE-14919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146922#comment-15146922 ] stack commented on HBASE-14919: --- [~eshcar] because of the above posted FAILURE? Nah, its not you I'd say [~eshcar] Our master branch is a little dodgy...flakey. See here, https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/ Looking over last few builds, I don't think any of the failures from this patch. Will let you know if I figure a connection (smile). Thanks for asking. > Infrastructure refactoring for In-Memory Flush > -- > > Key: HBASE-14919 > URL: https://issues.apache.org/jira/browse/HBASE-14919 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Fix For: 2.0.0 > > Attachments: HBASE-14919-V01.patch, HBASE-14919-V01.patch, > HBASE-14919-V02.patch, HBASE-14919-V03.patch, HBASE-14919-V04.patch, > HBASE-14919-V04.patch, HBASE-14919-V05.patch, HBASE-14919-V06.patch, > HBASE-14919-V06.patch, HBASE-14919-V07.patch, HBASE-14919-V08.patch, > HBASE-14919-V09.patch, HBASE-14919-V10.patch, HBASE-14919-V11.patch, > HBASE-14919-V12.patch > > > Refactoring the MemStore hierarchy, introducing segment (StoreSegment) as > first-class citizen and decoupling memstore scanner from the memstore > implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15259) WALEdits under replay will also be replicated
[
https://issues.apache.org/jira/browse/HBASE-15259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-15259:
--
Attachment: HBASE-15259.patch
Test locally, it is really a bug.
Here is my patch and testcase.
> WALEdits under replay will also be replicated
> -
>
> Key: HBASE-15259
> URL: https://issues.apache.org/jira/browse/HBASE-15259
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Attachments: HBASE-15259.patch
>
>
> I need to verify this but seeing the code
> {code}
> try {
> // We are about to append this edit; update the region-scoped
> sequence number. Do it
> // here inside this single appending/writing thread. Events are
> ordered on the ringbuffer
> // so region sequenceids will also be in order.
> regionSequenceId = entry.stampRegionSequenceId();
> // Edits are empty, there is nothing to append. Maybe empty when we
> are looking for a
> // region sequence id only, a region edit/sequence id that is not
> associated with an actual
> // edit. It has to go through all the rigmarole to be sure we have
> the right ordering.
> if (entry.getEdit().isEmpty()) {
> return;
> }
> // Coprocessor hook.
> if (!coprocessorHost.preWALWrite(entry.getHRegionInfo(),
> entry.getKey(),
> entry.getEdit())) {
> if (entry.getEdit().isReplay()) {
> // Set replication scope null so that this won't be replicated
> entry.getKey().setScopes(null);
> }
> }
> if (!listeners.isEmpty()) {
> for (WALActionsListener i: listeners) {
> // TODO: Why does listener take a table description and CPs take
> a regioninfo? Fix.
> i.visitLogEntryBeforeWrite(entry.getHTableDescriptor(),
> entry.getKey(),
> entry.getEdit());
> }
> }
> {code}
> When a WALEdit is in replay we set the Logkey to null. But in the
> visitLogEntryBeforeWrite() we again set the LogKey based on the replication
> scope associated with the cells. So the previous step of setting null does
> not work here?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15267) Read requests count metric is increased when Get operation does not return any record
[ https://issues.apache.org/jira/browse/HBASE-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15267: Attachment: HBASE-15267-v1.patch Test for scan not existing row is added. > Read requests count metric is increased when Get operation does not return > any record > - > > Key: HBASE-15267 > URL: https://issues.apache.org/jira/browse/HBASE-15267 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15267-v1.patch, HBASE-15267.patch > > > Read requests count is increased when Get operation returns no record by > filtering out. In such cases, Get for deleted record, Get for TTL expired > record and Get for filtering out. > So I fixed the bug and added some more test cases for read metrics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15267) Read requests count metric is increased when Get operation does not return any record
[ https://issues.apache.org/jira/browse/HBASE-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15267: Status: Patch Available (was: Open) > Read requests count metric is increased when Get operation does not return > any record > - > > Key: HBASE-15267 > URL: https://issues.apache.org/jira/browse/HBASE-15267 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15267-v1.patch, HBASE-15267.patch > > > Read requests count is increased when Get operation returns no record by > filtering out. In such cases, Get for deleted record, Get for TTL expired > record and Get for filtering out. > So I fixed the bug and added some more test cases for read metrics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15267) Read requests count metric is increased when Get operation does not return any record
[ https://issues.apache.org/jira/browse/HBASE-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15267: Status: Open (was: Patch Available) > Read requests count metric is increased when Get operation does not return > any record > - > > Key: HBASE-15267 > URL: https://issues.apache.org/jira/browse/HBASE-15267 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15267.patch > > > Read requests count is increased when Get operation returns no record by > filtering out. In such cases, Get for deleted record, Get for TTL expired > record and Get for filtering out. > So I fixed the bug and added some more test cases for read metrics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15267) Read requests count metric is increased when Get operation does not return any record
[
https://issues.apache.org/jira/browse/HBASE-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146897#comment-15146897
]
Hadoop QA commented on HBASE-15267:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
21m 35s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 4s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 16s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 202m 35s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests |
hadoop.hbase.master.TestMasterStatusServlet |
| JDK v1.7.0_95 Failed junit tests |
hadoop.hbase.master.TestMasterStatusServlet |
| JDK v1.7.0_95 Timed out junit tests |
org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-15 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12787880/HBASE-15267.patch |
| JIRA Issue | HBASE-15267 |
| Optional Tests | asflicense javac
[jira] [Updated] (HBASE-15129) Set default value for hbase.fs.tmp.dir rather than fully depend on hbase-default.xml
[
https://issues.apache.org/jira/browse/HBASE-15129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-15129:
--
Release Note: Before HBASE-15129, if somehow hbase-default.xml is not on
classpath, default values for hbase.fs.tmp.dir and hbase.bulkload.staging.dir
are left empty. After HBASE-15129, default values of both properties are set
to "/user//hbase-staging".
Sorry for the lag but I'm just back from the Spring Festival vacation, here is
the release note.
btw, Happy Chinese New Year to you all. [~ndimiduk] [~apurtell] [~busbey]
[~enis] :-)
> Set default value for hbase.fs.tmp.dir rather than fully depend on
> hbase-default.xml
>
>
> Key: HBASE-15129
> URL: https://issues.apache.org/jira/browse/HBASE-15129
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15129.patch, HBASE-15129_v2.patch
>
>
> One of our users has observed below error when our cluster upgrades from
> 0.98.12 to 1.1.2:
> {noformat}
> java.lang.IllegalArgumentException: Can not create a Path from a null string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:122)
> at org.apache.hadoop.fs.Path.(Path.java:134)
> at org.apache.hadoop.fs.Path.(Path.java:88)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configurePartitioner(HFileOutputFormat2.java:639)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configureIncrementalLoad(HFileOutputFormat2.java:489)
> {noformat}
> And checking the application code:
> {code}
> private Job createJob(Configuration jobConf) throws IOException {
> String tableName = jobConf.get(TABLE_NAME_KEY);
> Job job = new Job(jobConf);
> Configuration tableConf = HBaseConfiguration.create();
> HTable table = new HTable(tableConf, tableName);
> HFileOutputFormat2.configureIncrementalLoad(job, table);
> return job;
> }
> {code}
> We could see the user has his own hbase-independent job configuration, so our
> current code in HFileOutputFormat2:
> {code}
> Path partitionsPath = new Path(conf.get("hbase.fs.tmp.dir"), "partitions_" +
> UUID.randomUUID());
> {code}
> will return a null string and cause the above exception.
> We propose to add default value in the code rather than depend on
> hbase-default.xml in this JIRA.
> This is an improvement for HBASE-13625
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15267) Read requests count metric is increased when Get operation does not return any record
[ https://issues.apache.org/jira/browse/HBASE-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146884#comment-15146884 ] Eungsop Yoo commented on HBASE-15267: - Currently (without this patch) Scan for not existing row does not increase the read requests count, but Get for not existing row increase the read requests count. It seems to be inconsistent. How do you think about this? > Read requests count metric is increased when Get operation does not return > any record > - > > Key: HBASE-15267 > URL: https://issues.apache.org/jira/browse/HBASE-15267 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15267.patch > > > Read requests count is increased when Get operation returns no record by > filtering out. In such cases, Get for deleted record, Get for TTL expired > record and Get for filtering out. > So I fixed the bug and added some more test cases for read metrics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15267) Read requests count metric is increased when Get operation does not return any record
[ https://issues.apache.org/jira/browse/HBASE-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146880#comment-15146880 ] Eungsop Yoo commented on HBASE-15267: - OK. My explanation was not enough. My intention was that how many gets were issued that returning no record is countered as filtered read requests metric, which was add in [HBASE-15197|https://issues.apache.org/jira/browse/HBASE-15197]. But in current patch, the requests for not existing rows is not countered at all. I'll try to count it as filtered read requests metric. How about this? > Read requests count metric is increased when Get operation does not return > any record > - > > Key: HBASE-15267 > URL: https://issues.apache.org/jira/browse/HBASE-15267 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15267.patch > > > Read requests count is increased when Get operation returns no record by > filtering out. In such cases, Get for deleted record, Get for TTL expired > record and Get for filtering out. > So I fixed the bug and added some more test cases for read metrics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15267) Read requests count metric is increased when Get operation does not return any record
[ https://issues.apache.org/jira/browse/HBASE-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146829#comment-15146829 ] Elliott Clark commented on HBASE-15267: --- Seems like the metric is correct. It's all about how many gets were issued. It doesn't say gets that returned things. Just number of gets. The get could have done a lot of IO so we shouldn't ignore that. > Read requests count metric is increased when Get operation does not return > any record > - > > Key: HBASE-15267 > URL: https://issues.apache.org/jira/browse/HBASE-15267 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15267.patch > > > Read requests count is increased when Get operation returns no record by > filtering out. In such cases, Get for deleted record, Get for TTL expired > record and Get for filtering out. > So I fixed the bug and added some more test cases for read metrics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15267) Read requests count metric is increased when Get operation does not return any record
[ https://issues.apache.org/jira/browse/HBASE-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15267: Status: Patch Available (was: Open) > Read requests count metric is increased when Get operation does not return > any record > - > > Key: HBASE-15267 > URL: https://issues.apache.org/jira/browse/HBASE-15267 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15267.patch > > > Read requests count is increased when Get operation returns no record by > filtering out. In such cases, Get for deleted record, Get for TTL expired > record and Get for filtering out. > So I fixed the bug and added some more test cases for read metrics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15267) Read requests count metric is increased when Get operation does not return any record
[ https://issues.apache.org/jira/browse/HBASE-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15267: Attachment: HBASE-15267.patch > Read requests count metric is increased when Get operation does not return > any record > - > > Key: HBASE-15267 > URL: https://issues.apache.org/jira/browse/HBASE-15267 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15267.patch > > > Read requests count is increased when Get operation returns no record by > filtering out. In such cases, Get for deleted record, Get for TTL expired > record and Get for filtering out. > So I fixed the bug and added some more test cases for read metrics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15267) Read requests count metric is increased when Get operation does not return any record
Eungsop Yoo created HBASE-15267: --- Summary: Read requests count metric is increased when Get operation does not return any record Key: HBASE-15267 URL: https://issues.apache.org/jira/browse/HBASE-15267 Project: HBase Issue Type: Bug Reporter: Eungsop Yoo Priority: Minor Read requests count is increased when Get operation returns no record by filtering out. In such cases, Get for deleted record, Get for TTL expired record and Get for filtering out. So I fixed the bug and added some more test cases for read metrics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15180) Reduce garbage created while reading Cells from Codec Decoder
[
https://issues.apache.org/jira/browse/HBASE-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146768#comment-15146768
]
Yu Li commented on HBASE-15180:
---
Below change in the v4 patch seems like some typo
{code}
+import com.sun.xml.internal.rngom.parse.compact.EOFException
{code}
and we should change to use {{java.io.EOFException}}? [~anoop.hbase]
> Reduce garbage created while reading Cells from Codec Decoder
> -
>
> Key: HBASE-15180
> URL: https://issues.apache.org/jira/browse/HBASE-15180
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 0.98.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0, 1.3.0, 1.2.1
>
> Attachments: HBASE-15180.patch, HBASE-15180_V2.patch,
> HBASE-15180_V4.patch
>
>
> In KeyValueDecoder#parseCell (Default Codec decoder) we use
> KeyValueUtil#iscreate to read cells from the InputStream. Here we 1st create
> a byte[] of length 4 and read the cell length and then an array of Cell's
> length and read in cell bytes into it and create a KV.
> Actually in server we read the reqs into a byte[] and CellScanner is created
> on top of a ByteArrayInputStream on top of this. By default in write path, we
> have MSLAB usage ON. So while adding Cells to memstore, we will copy the Cell
> bytes to MSLAB memory chunks (default 2 MB size) and recreate Cells over that
> bytes. So there is no issue if we create Cells over the RPC read byte[]
> directly here in Decoder. No need for 2 byte[] creation and copy for every
> Cell in request.
> My plan is to make a Cell aware ByteArrayInputStream which can read Cells
> directly from it.
> Same Codec path is used in client side also. There better we can avoid this
> direct Cell create and continue to do the copy to smaller byte[]s path. Plan
> to introduce some thing like a CodecContext associated with every Codec
> instance which can say the server/client context.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15219) Canary tool does not return non-zero exit code when one of regions is in stuck state
[
https://issues.apache.org/jira/browse/HBASE-15219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146769#comment-15146769
]
Ted Yu commented on HBASE-15219:
TestMasterStatusServlet failure likely was caused by HBASE-13839
> Canary tool does not return non-zero exit code when one of regions is in
> stuck state
> -
>
> Key: HBASE-15219
> URL: https://issues.apache.org/jira/browse/HBASE-15219
> Project: HBase
> Issue Type: Improvement
> Components: canary
>Affects Versions: 0.98.16
>Reporter: Vishal Khandelwal
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 0.98.18
>
> Attachments: HBASE-15219-branch-1.2.v8.patch, HBASE-15219.v1.patch,
> HBASE-15219.v3.patch, HBASE-15219.v4.patch, HBASE-15219.v5.patch,
> HBASE-15219.v7.patch, HBASE-15219.v8.patch, HBASE-15219.v9.patch
>
>
> {code}
> 2016-02-05 12:24:18,571 ERROR [pool-2-thread-7] tool.Canary - read from
> region
> CAN_1,\x08\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1454667477865.00e77d07b8defe10704417fb99aa0418.
> column family 0 failed
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=2, exceptions:
> Fri Feb 05 12:24:15 GMT 2016,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@54c9fea0,
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> CAN_1,\x08\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1454667477865.00e77d07b8defe10704417fb99aa0418.
> is not online on isthbase02-dnds1-3-crd.eng.sfdc.net,60020,1454669984738
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2852)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4468)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2984)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31186)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2149)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:104)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
>
> -bash-4.1$ echo $?
> 0
> {code}
> Below code prints the error but it does sets/returns the exit code. Due to
> this tool can't be integrated with nagios or other alerting.
> Ideally it should return error for failures. as pre the documentation:
>
> This tool will return non zero error codes to user for collaborating with
> other monitoring tools, such as Nagios. The error code definitions are:
> private static final int USAGE_EXIT_CODE = 1;
> private static final int INIT_ERROR_EXIT_CODE = 2;
> private static final int TIMEOUT_ERROR_EXIT_CODE = 3;
> private static final int ERROR_EXIT_CODE = 4;
>
> {code}
> org.apache.hadoop.hbase.tool.Canary.RegionTask
> public Void read() {
>
> try {
> table = connection.getTable(region.getTable());
> tableDesc = table.getTableDescriptor();
> } catch (IOException e) {
> LOG.debug("sniffRegion failed", e);
> sink.publishReadFailure(region, e);
>...
> return null;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15266) add precommit check for "catch Throwable"
[ https://issues.apache.org/jira/browse/HBASE-15266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146712#comment-15146712 ] Sean Busbey commented on HBASE-15266: - Exceptions to a rule are likely to exist, and the judgement of our committers should be able to handle identifying appropriate uses and ensuring that we're handling things correctly (for example making sure we're not discarding Errors). > add precommit check for "catch Throwable" > - > > Key: HBASE-15266 > URL: https://issues.apache.org/jira/browse/HBASE-15266 > Project: HBase > Issue Type: New Feature > Components: test >Reporter: Sean Busbey >Priority: Minor > > Catching Throwable is usually incorrect because it gets all of the Error > derived problems, like ThreadDeath, OutOfMemoryError, VM malfunctions, linker > problems, etc. > I was surprised to see findbugs did not flag catching throwable in the > patches on HBASE-9393. We should add a precommit check that looks expressly > for it, in our "hbase antipatterns" test plugin. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15261) Make Throwable t in DaughterOpener volatile
[ https://issues.apache.org/jira/browse/HBASE-15261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146644#comment-15146644 ] huaxiang sun commented on HBASE-15261: -- The exception is same as http://stackoverflow.com/questions/34905296/hbase-periodicmemstoreflusher-exception There is no logs so it is hard to tell the root cause. > Make Throwable t in DaughterOpener volatile > --- > > Key: HBASE-15261 > URL: https://issues.apache.org/jira/browse/HBASE-15261 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Attachments: HBASE-15261-001.patch > > > In the region split process, daughter regions are opened in different > threads, Throwable t is set in these threads and it is checked in the calling > thread. Need to make it volatile so the checking will not miss any exceptions > from opening daughter regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14949) Resolve name conflict when splitting if there are duplicated WAL entries
[
https://issues.apache.org/jira/browse/HBASE-14949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146438#comment-15146438
]
Hadoop QA commented on HBASE-14949:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 27s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 111m 43s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_72.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 90m 52s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 251m 15s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests |
hadoop.hbase.master.TestMasterStatusServlet |
| JDK v1.7.0_95 Failed junit tests |
hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster |
| | hadoop.hbase.master.TestMasterStatusServlet |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-14 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12787852/HBASE-14949-v4.patch |
| JIRA Issue | HBASE-14949 |
| Optional Tests | asflicense javac javadoc unit findbugs
[jira] [Commented] (HBASE-15263) TestIPv6NIOServerSocketChannel.testServerSocketFromLocalhostResolution can hang indefinetly
[
https://issues.apache.org/jira/browse/HBASE-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146471#comment-15146471
]
Heng Chen commented on HBASE-15263:
---
[~stack] Thread is in RUNNABLE state?
> TestIPv6NIOServerSocketChannel.testServerSocketFromLocalhostResolution can
> hang indefinetly
> ---
>
> Key: HBASE-15263
> URL: https://issues.apache.org/jira/browse/HBASE-15263
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>
> Should have a timeout on this test at least. Stuck here:
> {code}
> "main" #1 prio=5 os_prio=0 tid=0x7f3f9c008000 nid=0xca97 runnable
> [0x7f3fa2e8c000]
>java.lang.Thread.State: RUNNABLE
> at
> org.apache.hadoop.hbase.HBaseTestingUtility$PortAllocator.randomFreePort(HBaseTestingUtility.java:3500)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.randomFreePort(HBaseTestingUtility.java:3450)
> at
> org.apache.hadoop.hbase.TestIPv6NIOServerSocketChannel.bindServerSocket(TestIPv6NIOServerSocketChannel.java:57)
> at
> org.apache.hadoop.hbase.TestIPv6NIOServerSocketChannel.testServerSocketFromLocalhostResolution(TestIPv6NIOServerSocketChannel.java:151)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:108)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:78)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:54)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:144)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14919) Infrastructure refactoring for In-Memory Flush
[ https://issues.apache.org/jira/browse/HBASE-14919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146472#comment-15146472 ] Eshcar Hillel commented on HBASE-14919: --- [~stack] I see we have some problems with the commit. Anything I can do to help? > Infrastructure refactoring for In-Memory Flush > -- > > Key: HBASE-14919 > URL: https://issues.apache.org/jira/browse/HBASE-14919 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Fix For: 2.0.0 > > Attachments: HBASE-14919-V01.patch, HBASE-14919-V01.patch, > HBASE-14919-V02.patch, HBASE-14919-V03.patch, HBASE-14919-V04.patch, > HBASE-14919-V04.patch, HBASE-14919-V05.patch, HBASE-14919-V06.patch, > HBASE-14919-V06.patch, HBASE-14919-V07.patch, HBASE-14919-V08.patch, > HBASE-14919-V09.patch, HBASE-14919-V10.patch, HBASE-14919-V11.patch, > HBASE-14919-V12.patch > > > Refactoring the MemStore hierarchy, introducing segment (StoreSegment) as > first-class citizen and decoupling memstore scanner from the memstore > implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15264) Implement a fan out HDFS OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-15264: -- Attachment: HBASE-15264.patch FanOutOneBlockAsyncDFSOutput implementation. > Implement a fan out HDFS OutputStream > - > > Key: HBASE-15264 > URL: https://issues.apache.org/jira/browse/HBASE-15264 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang > Attachments: HBASE-15264.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15264) Implement a fan out HDFS OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-15264: -- Status: Patch Available (was: Open) > Implement a fan out HDFS OutputStream > - > > Key: HBASE-15264 > URL: https://issues.apache.org/jira/browse/HBASE-15264 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang > Attachments: HBASE-15264.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15266) add precommit check for "catch Throwable"
[ https://issues.apache.org/jira/browse/HBASE-15266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146456#comment-15146456 ] Heng Chen commented on HBASE-15266: --- IMO some monitor threads need to catch throwable. > add precommit check for "catch Throwable" > - > > Key: HBASE-15266 > URL: https://issues.apache.org/jira/browse/HBASE-15266 > Project: HBase > Issue Type: New Feature > Components: test >Reporter: Sean Busbey >Priority: Minor > > Catching Throwable is usually incorrect because it gets all of the Error > derived problems, like ThreadDeath, OutOfMemoryError, VM malfunctions, linker > problems, etc. > I was surprised to see findbugs did not flag catching throwable in the > patches on HBASE-9393. We should add a precommit check that looks expressly > for it, in our "hbase antipatterns" test plugin. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[
https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146507#comment-15146507
]
Hadoop QA commented on HBASE-15264:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 15s
{color} | {color:red} Patch generated 22 new checkstyle issues in hbase-server
(total was 0, now 22). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 50s
{color} | {color:red} Patch causes 65 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 39s
{color} | {color:red} Patch causes 65 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 28s
{color} | {color:red} Patch causes 65 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 19s
{color} | {color:red} Patch causes 65 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 8s
{color} | {color:red} Patch causes 65 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 59s
{color} | {color:red} Patch causes 59 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 49s
{color} | {color:red} Patch causes 59 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 38s
{color} | {color:red} Patch causes 59 errors with Hadoop v2.6.3. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 2m 26s
{color} | {color:red} hbase-server-jdk1.8.0_72 with JDK v1.8.0_72 generated 1
new issues (was 1, now 2). {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} |
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[
https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146518#comment-15146518
]
Duo Zhang commented on HBASE-15264:
---
Let me try addressing the hadoop version issue...
For checkstyle, "total was 0", is it possible? And yeah I could fix some, but
this one
{noformat}
./hbase-server/src/main/java/org/apache/hadoop/hbase/util/FanOutOneBlockAsyncDFSOutput.java:159:33:
error: Inner assignments should be avoided.
{noformat}
{code:title=FanOutOneBlockAsyncDFSOutput.java}
for (Callback cb; (cb = waitingAckQueue.peekFirst()) != null;) {
{code}
I think this is a standard pattern? How do I fix this...
Thanks.
> Implement a fan out HDFS OutputStream
> -
>
> Key: HBASE-15264
> URL: https://issues.apache.org/jira/browse/HBASE-15264
> Project: HBase
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Attachments: HBASE-15264.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15266) add precommit check for "catch Throwable"
[ https://issues.apache.org/jira/browse/HBASE-15266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146445#comment-15146445 ] Anoop Sam John commented on HBASE-15266: I have seen many a places in our code base, we have catch of Throwable.. > add precommit check for "catch Throwable" > - > > Key: HBASE-15266 > URL: https://issues.apache.org/jira/browse/HBASE-15266 > Project: HBase > Issue Type: New Feature > Components: test >Reporter: Sean Busbey >Priority: Minor > > Catching Throwable is usually incorrect because it gets all of the Error > derived problems, like ThreadDeath, OutOfMemoryError, VM malfunctions, linker > problems, etc. > I was surprised to see findbugs did not flag catching throwable in the > patches on HBASE-9393. We should add a precommit check that looks expressly > for it, in our "hbase antipatterns" test plugin. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14949) Resolve name conflict when splitting if there are duplicated WAL entries
[
https://issues.apache.org/jira/browse/HBASE-14949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146452#comment-15146452
]
Duo Zhang commented on HBASE-14949:
---
[~stack] This is needed by the {{AsyncFSHLog}}, PTAL.
Thanks.
> Resolve name conflict when splitting if there are duplicated WAL entries
>
>
> Key: HBASE-14949
> URL: https://issues.apache.org/jira/browse/HBASE-14949
> Project: HBase
> Issue Type: Sub-task
>Reporter: Heng Chen
>Assignee: Duo Zhang
> Attachments: HBASE-14949-v3.patch, HBASE-14949-v4.patch,
> HBASE-14949.patch, HBASE-14949_v1.patch, HBASE-14949_v2.patch
>
>
> The AsyncFSHLog introduced in HBASE-14790 may write same WAL entries to
> different WAL files. WAL entry itself is idempotent so replay is not a
> problem but the intermediate file name and final name when splitting is
> constructed using the lowest or highest sequence id of the WAL entries
> written, so it is possible that different WAL files will have same
> intermediate or final file name when splitting. In the currentm
> implementation, this will cause split fail or data loss. We need to solve
> this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
