[jira] [Commented] (HBASE-21824) change master and RS UI links to be relative
[ https://issues.apache.org/jira/browse/HBASE-21824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758687#comment-16758687 ] Tommy Li commented on HBASE-21824: -- +1 > change master and RS UI links to be relative > > > Key: HBASE-21824 > URL: https://issues.apache.org/jira/browse/HBASE-21824 > Project: HBase > Issue Type: Improvement >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HBASE-21824.patch > > > When HBase services are accessed thru the proxy e.g. > proxy/foo/bar/machine:port/master-status, the current links on the page lead > to e.g. proxy/procedures.jsp, because they start with a slash. There seems to > be no reason for them to have a slash since all the pages are on the same > level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21775) The BufferedMutator doesn't ever refresh region location cache
[
https://issues.apache.org/jira/browse/HBASE-21775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756687#comment-16756687
]
Tommy Li commented on HBASE-21775:
--
Sorry about that [~stack], yeah I need to update my editor settings to match
this project's styleguide
> The BufferedMutator doesn't ever refresh region location cache
> --
>
> Key: HBASE-21775
> URL: https://issues.apache.org/jira/browse/HBASE-21775
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Tommy Li
>Assignee: Tommy Li
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.1.3, 2.0.5, 1.3.4
>
> Attachments: HBASE-21775-ADDENDUM.master.001.patch,
> HBASE-21775.master.001.patch,
> org.apache.hadoop.hbase.client.TestAsyncProcess-with-HBASE-21775.txt,
> org.apache.hadoop.hbase.client.TestAsyncProcess-without-HBASE-21775.txt
>
>
> {color:#22}I noticed in some of my writing jobs that the BufferedMutator
> would get stuck retrying writes against a dead server.{color}
> {code:java}
> 19/01/18 15:15:47 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:15:54 WARN [htable-pool3-t56] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=15/21, failureCount=1ops, last
> exception=org.apache.hadoop.hbase.DoNotRetryIOException: Operation rpcTimeout
> on ,17020,1547848193782, tracking started Fri Jan 18 14:55:37 PST
> 2019; NOT retrying, failed=1 -- final attempt!
> 19/01/18 15:15:54 ERROR [Executor task launch worker for task 0]
> IngestRawData.map(): [B@258bc2c7:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: Operation rpcTimeout: 1 time, servers with issues:
> ,17020,1547848193782
> {code}
>
> After the single remaining action permanently failed, it would resume
> progress only to get stuck again retrying against the same dead server:
> {code:java}
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:20 INFO [htable-pool3-t55] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=6/21, failureCount=1ops, last
> exception=java.net.ConnectException: Call to failed on connection
> exception:
> org.apache.hbase.thirdparty.io.netty.channel.ConnectTimeoutException:
> connection timed out: on ,17020,1547848193782, tracking
> started null, retrying after=20089ms, operationsToReplay=1
> {code}
>
> Only restarting the client process to generate a new BufferedMutator instance
> would fix the issue, at least until the next regionserver crash
> The logs I've pasted show the issue happening with a
> ConnectionTimeoutException, but we've also seen it with
> NotServingRegionException and some others
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21775) The BufferedMutator doesn't ever refresh region location cache
[
https://issues.apache.org/jira/browse/HBASE-21775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tommy Li updated HBASE-21775:
-
Status: Patch Available (was: Reopened)
> The BufferedMutator doesn't ever refresh region location cache
> --
>
> Key: HBASE-21775
> URL: https://issues.apache.org/jira/browse/HBASE-21775
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Tommy Li
>Assignee: Tommy Li
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.1.3, 2.0.5, 1.3.4
>
> Attachments: HBASE-21775-ADDENDUM.master.001.patch,
> HBASE-21775.master.001.patch,
> org.apache.hadoop.hbase.client.TestAsyncProcess-with-HBASE-21775.txt,
> org.apache.hadoop.hbase.client.TestAsyncProcess-without-HBASE-21775.txt
>
>
> {color:#22}I noticed in some of my writing jobs that the BufferedMutator
> would get stuck retrying writes against a dead server.{color}
> {code:java}
> 19/01/18 15:15:47 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:15:54 WARN [htable-pool3-t56] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=15/21, failureCount=1ops, last
> exception=org.apache.hadoop.hbase.DoNotRetryIOException: Operation rpcTimeout
> on ,17020,1547848193782, tracking started Fri Jan 18 14:55:37 PST
> 2019; NOT retrying, failed=1 -- final attempt!
> 19/01/18 15:15:54 ERROR [Executor task launch worker for task 0]
> IngestRawData.map(): [B@258bc2c7:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: Operation rpcTimeout: 1 time, servers with issues:
> ,17020,1547848193782
> {code}
>
> After the single remaining action permanently failed, it would resume
> progress only to get stuck again retrying against the same dead server:
> {code:java}
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:20 INFO [htable-pool3-t55] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=6/21, failureCount=1ops, last
> exception=java.net.ConnectException: Call to failed on connection
> exception:
> org.apache.hbase.thirdparty.io.netty.channel.ConnectTimeoutException:
> connection timed out: on ,17020,1547848193782, tracking
> started null, retrying after=20089ms, operationsToReplay=1
> {code}
>
> Only restarting the client process to generate a new BufferedMutator instance
> would fix the issue, at least until the next regionserver crash
> The logs I've pasted show the issue happening with a
> ConnectionTimeoutException, but we've also seen it with
> NotServingRegionException and some others
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21775) The BufferedMutator doesn't ever refresh region location cache
[
https://issues.apache.org/jira/browse/HBASE-21775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756608#comment-16756608
]
Tommy Li commented on HBASE-21775:
--
So what I noticed is that those five tests only fail when I run the entire test
class, but running them individually always succeeds. The class has a static
configuration that gets modified by some tests. When I changed the tests to
create a new configuration per test, the tests stopped failing on my machine.
I've uploaded HBASE-21775-ADDENDUM.master.001.patch with my changes
> The BufferedMutator doesn't ever refresh region location cache
> --
>
> Key: HBASE-21775
> URL: https://issues.apache.org/jira/browse/HBASE-21775
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Tommy Li
>Assignee: Tommy Li
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.1.3, 2.0.5, 1.3.4
>
> Attachments: HBASE-21775-ADDENDUM.master.001.patch,
> HBASE-21775.master.001.patch,
> org.apache.hadoop.hbase.client.TestAsyncProcess-with-HBASE-21775.txt,
> org.apache.hadoop.hbase.client.TestAsyncProcess-without-HBASE-21775.txt
>
>
> {color:#22}I noticed in some of my writing jobs that the BufferedMutator
> would get stuck retrying writes against a dead server.{color}
> {code:java}
> 19/01/18 15:15:47 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:15:54 WARN [htable-pool3-t56] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=15/21, failureCount=1ops, last
> exception=org.apache.hadoop.hbase.DoNotRetryIOException: Operation rpcTimeout
> on ,17020,1547848193782, tracking started Fri Jan 18 14:55:37 PST
> 2019; NOT retrying, failed=1 -- final attempt!
> 19/01/18 15:15:54 ERROR [Executor task launch worker for task 0]
> IngestRawData.map(): [B@258bc2c7:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: Operation rpcTimeout: 1 time, servers with issues:
> ,17020,1547848193782
> {code}
>
> After the single remaining action permanently failed, it would resume
> progress only to get stuck again retrying against the same dead server:
> {code:java}
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:20 INFO [htable-pool3-t55] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=6/21, failureCount=1ops, last
> exception=java.net.ConnectException: Call to failed on connection
> exception:
> org.apache.hbase.thirdparty.io.netty.channel.ConnectTimeoutException:
> connection timed out: on ,17020,1547848193782, tracking
> started null, retrying after=20089ms, operationsToReplay=1
> {code}
>
> Only restarting the client process to generate a new BufferedMutator instance
> would fix the issue, at least until the next regionserver crash
> The logs I've pasted show the issue happening with a
> ConnectionTimeoutException, but we've also seen it with
> NotServingRegionException and some others
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21775) The BufferedMutator doesn't ever refresh region location cache
[
https://issues.apache.org/jira/browse/HBASE-21775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tommy Li updated HBASE-21775:
-
Attachment: HBASE-21775-ADDENDUM.master.001.patch
> The BufferedMutator doesn't ever refresh region location cache
> --
>
> Key: HBASE-21775
> URL: https://issues.apache.org/jira/browse/HBASE-21775
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Tommy Li
>Assignee: Tommy Li
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.1.3, 2.0.5, 1.3.4
>
> Attachments: HBASE-21775-ADDENDUM.master.001.patch,
> HBASE-21775.master.001.patch,
> org.apache.hadoop.hbase.client.TestAsyncProcess-with-HBASE-21775.txt,
> org.apache.hadoop.hbase.client.TestAsyncProcess-without-HBASE-21775.txt
>
>
> {color:#22}I noticed in some of my writing jobs that the BufferedMutator
> would get stuck retrying writes against a dead server.{color}
> {code:java}
> 19/01/18 15:15:47 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:15:54 WARN [htable-pool3-t56] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=15/21, failureCount=1ops, last
> exception=org.apache.hadoop.hbase.DoNotRetryIOException: Operation rpcTimeout
> on ,17020,1547848193782, tracking started Fri Jan 18 14:55:37 PST
> 2019; NOT retrying, failed=1 -- final attempt!
> 19/01/18 15:15:54 ERROR [Executor task launch worker for task 0]
> IngestRawData.map(): [B@258bc2c7:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: Operation rpcTimeout: 1 time, servers with issues:
> ,17020,1547848193782
> {code}
>
> After the single remaining action permanently failed, it would resume
> progress only to get stuck again retrying against the same dead server:
> {code:java}
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:20 INFO [htable-pool3-t55] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=6/21, failureCount=1ops, last
> exception=java.net.ConnectException: Call to failed on connection
> exception:
> org.apache.hbase.thirdparty.io.netty.channel.ConnectTimeoutException:
> connection timed out: on ,17020,1547848193782, tracking
> started null, retrying after=20089ms, operationsToReplay=1
> {code}
>
> Only restarting the client process to generate a new BufferedMutator instance
> would fix the issue, at least until the next regionserver crash
> The logs I've pasted show the issue happening with a
> ConnectionTimeoutException, but we've also seen it with
> NotServingRegionException and some others
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21775) The BufferedMutator doesn't ever refresh region location cache
[
https://issues.apache.org/jira/browse/HBASE-21775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756436#comment-16756436
]
Tommy Li commented on HBASE-21775:
--
I was able to get a somewhat reliable local repro - will take a look later today
> The BufferedMutator doesn't ever refresh region location cache
> --
>
> Key: HBASE-21775
> URL: https://issues.apache.org/jira/browse/HBASE-21775
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Tommy Li
>Assignee: Tommy Li
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.1.3, 2.0.5, 1.3.4
>
> Attachments: HBASE-21775.master.001.patch,
> org.apache.hadoop.hbase.client.TestAsyncProcess-with-HBASE-21775.txt,
> org.apache.hadoop.hbase.client.TestAsyncProcess-without-HBASE-21775.txt
>
>
> {color:#22}I noticed in some of my writing jobs that the BufferedMutator
> would get stuck retrying writes against a dead server.{color}
> {code:java}
> 19/01/18 15:15:47 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:15:54 WARN [htable-pool3-t56] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=15/21, failureCount=1ops, last
> exception=org.apache.hadoop.hbase.DoNotRetryIOException: Operation rpcTimeout
> on ,17020,1547848193782, tracking started Fri Jan 18 14:55:37 PST
> 2019; NOT retrying, failed=1 -- final attempt!
> 19/01/18 15:15:54 ERROR [Executor task launch worker for task 0]
> IngestRawData.map(): [B@258bc2c7:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: Operation rpcTimeout: 1 time, servers with issues:
> ,17020,1547848193782
> {code}
>
> After the single remaining action permanently failed, it would resume
> progress only to get stuck again retrying against the same dead server:
> {code:java}
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:20 INFO [htable-pool3-t55] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=6/21, failureCount=1ops, last
> exception=java.net.ConnectException: Call to failed on connection
> exception:
> org.apache.hbase.thirdparty.io.netty.channel.ConnectTimeoutException:
> connection timed out: on ,17020,1547848193782, tracking
> started null, retrying after=20089ms, operationsToReplay=1
> {code}
>
> Only restarting the client process to generate a new BufferedMutator instance
> would fix the issue, at least until the next regionserver crash
> The logs I've pasted show the issue happening with a
> ConnectionTimeoutException, but we've also seen it with
> NotServingRegionException and some others
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21775) The BufferedMutator doesn't ever refresh region location cache
[
https://issues.apache.org/jira/browse/HBASE-21775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755347#comment-16755347
]
Tommy Li commented on HBASE-21775:
--
Thanks for the link, [~stack]. I took a look at the report from before my
change went in and indeed TestAsyncProcess [is not listed
there|[https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.1/168/artifact/dashboard.html].]
Could this be a build caching issue?
> The BufferedMutator doesn't ever refresh region location cache
> --
>
> Key: HBASE-21775
> URL: https://issues.apache.org/jira/browse/HBASE-21775
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Tommy Li
>Assignee: Tommy Li
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.1.3, 2.0.5, 1.3.4
>
> Attachments: HBASE-21775.master.001.patch,
> org.apache.hadoop.hbase.client.TestAsyncProcess-with-HBASE-21775.txt,
> org.apache.hadoop.hbase.client.TestAsyncProcess-without-HBASE-21775.txt
>
>
> {color:#22}I noticed in some of my writing jobs that the BufferedMutator
> would get stuck retrying writes against a dead server.{color}
> {code:java}
> 19/01/18 15:15:47 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:15:54 WARN [htable-pool3-t56] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=15/21, failureCount=1ops, last
> exception=org.apache.hadoop.hbase.DoNotRetryIOException: Operation rpcTimeout
> on ,17020,1547848193782, tracking started Fri Jan 18 14:55:37 PST
> 2019; NOT retrying, failed=1 -- final attempt!
> 19/01/18 15:15:54 ERROR [Executor task launch worker for task 0]
> IngestRawData.map(): [B@258bc2c7:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: Operation rpcTimeout: 1 time, servers with issues:
> ,17020,1547848193782
> {code}
>
> After the single remaining action permanently failed, it would resume
> progress only to get stuck again retrying against the same dead server:
> {code:java}
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:20 INFO [htable-pool3-t55] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=6/21, failureCount=1ops, last
> exception=java.net.ConnectException: Call to failed on connection
> exception:
> org.apache.hbase.thirdparty.io.netty.channel.ConnectTimeoutException:
> connection timed out: on ,17020,1547848193782, tracking
> started null, retrying after=20089ms, operationsToReplay=1
> {code}
>
> Only restarting the client process to generate a new BufferedMutator instance
> would fix the issue, at least until the next regionserver crash
> The logs I've pasted show the issue happening with a
> ConnectionTimeoutException, but we've also seen it with
> NotServingRegionException and some others
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21775) The BufferedMutator doesn't ever refresh region location cache
[
https://issues.apache.org/jira/browse/HBASE-21775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755347#comment-16755347
]
Tommy Li edited comment on HBASE-21775 at 1/29/19 8:13 PM:
---
Thanks for the link, [~stack]. I took a look at the report from before my
change went in and indeed TestAsyncProcess [is not listed
there|https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.1/168/artifact/dashboard.html]
.Could this be a build caching issue?
was (Author: tommyzli):
Thanks for the link, [~stack]. I took a look at the report from before my
change went in and indeed TestAsyncProcess [is not listed
there|[https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.1/168/artifact/dashboard.html].]
Could this be a build caching issue?
> The BufferedMutator doesn't ever refresh region location cache
> --
>
> Key: HBASE-21775
> URL: https://issues.apache.org/jira/browse/HBASE-21775
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Tommy Li
>Assignee: Tommy Li
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.1.3, 2.0.5, 1.3.4
>
> Attachments: HBASE-21775.master.001.patch,
> org.apache.hadoop.hbase.client.TestAsyncProcess-with-HBASE-21775.txt,
> org.apache.hadoop.hbase.client.TestAsyncProcess-without-HBASE-21775.txt
>
>
> {color:#22}I noticed in some of my writing jobs that the BufferedMutator
> would get stuck retrying writes against a dead server.{color}
> {code:java}
> 19/01/18 15:15:47 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:15:54 WARN [htable-pool3-t56] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=15/21, failureCount=1ops, last
> exception=org.apache.hadoop.hbase.DoNotRetryIOException: Operation rpcTimeout
> on ,17020,1547848193782, tracking started Fri Jan 18 14:55:37 PST
> 2019; NOT retrying, failed=1 -- final attempt!
> 19/01/18 15:15:54 ERROR [Executor task launch worker for task 0]
> IngestRawData.map(): [B@258bc2c7:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: Operation rpcTimeout: 1 time, servers with issues:
> ,17020,1547848193782
> {code}
>
> After the single remaining action permanently failed, it would resume
> progress only to get stuck again retrying against the same dead server:
> {code:java}
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:20 INFO [htable-pool3-t55] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=6/21, failureCount=1ops, last
> exception=java.net.ConnectException: Call to failed on connection
> exception:
> org.apache.hbase.thirdparty.io.netty.channel.ConnectTimeoutException:
> connection timed out: on ,17020,1547848193782, tracking
> started null, retrying after=20089ms, operationsToReplay=1
> {code}
>
> Only restarting the client process to generate a new BufferedMutator instance
> would fix the issue, at least until the next regionserver crash
> The logs I've pasted show the issue happening with a
> ConnectionTimeoutException, but we've also seen it with
> NotServingRegionException and some others
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21775) The BufferedMutator doesn't ever refresh region location cache
[
https://issues.apache.org/jira/browse/HBASE-21775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755342#comment-16755342
]
Tommy Li commented on HBASE-21775:
--
So I pulled branch-2.1 and ran `mvn test
-Dtest=org.apache.hadoop.hbase.client.TestAsyncProcess
-Dskip.license.check=true` locally both with my change and without, and I see
the same 5 test failures in both runs. I've attached the surefire output of
both runs.
[^org.apache.hadoop.hbase.client.TestAsyncProcess-without-HBASE-21775.txt] has
one extra failure which is the test that I added.
Unless I'm looking at the wrong tests, I don't think the failures are
introduced by my change
> The BufferedMutator doesn't ever refresh region location cache
> --
>
> Key: HBASE-21775
> URL: https://issues.apache.org/jira/browse/HBASE-21775
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Tommy Li
>Assignee: Tommy Li
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.1.3, 2.0.5, 1.3.4
>
> Attachments: HBASE-21775.master.001.patch,
> org.apache.hadoop.hbase.client.TestAsyncProcess-with-HBASE-21775.txt,
> org.apache.hadoop.hbase.client.TestAsyncProcess-without-HBASE-21775.txt
>
>
> {color:#22}I noticed in some of my writing jobs that the BufferedMutator
> would get stuck retrying writes against a dead server.{color}
> {code:java}
> 19/01/18 15:15:47 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:15:54 WARN [htable-pool3-t56] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=15/21, failureCount=1ops, last
> exception=org.apache.hadoop.hbase.DoNotRetryIOException: Operation rpcTimeout
> on ,17020,1547848193782, tracking started Fri Jan 18 14:55:37 PST
> 2019; NOT retrying, failed=1 -- final attempt!
> 19/01/18 15:15:54 ERROR [Executor task launch worker for task 0]
> IngestRawData.map(): [B@258bc2c7:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: Operation rpcTimeout: 1 time, servers with issues:
> ,17020,1547848193782
> {code}
>
> After the single remaining action permanently failed, it would resume
> progress only to get stuck again retrying against the same dead server:
> {code:java}
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:20 INFO [htable-pool3-t55] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=6/21, failureCount=1ops, last
> exception=java.net.ConnectException: Call to failed on connection
> exception:
> org.apache.hbase.thirdparty.io.netty.channel.ConnectTimeoutException:
> connection timed out: on ,17020,1547848193782, tracking
> started null, retrying after=20089ms, operationsToReplay=1
> {code}
>
> Only restarting the client process to generate a new BufferedMutator instance
> would fix the issue, at least until the next regionserver crash
> The logs I've pasted show the issue happening with a
> ConnectionTimeoutException, but we've also seen it with
> NotServingRegionException and some others
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21775) The BufferedMutator doesn't ever refresh region location cache
[
https://issues.apache.org/jira/browse/HBASE-21775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tommy Li updated HBASE-21775:
-
Attachment:
org.apache.hadoop.hbase.client.TestAsyncProcess-without-HBASE-21775.txt
org.apache.hadoop.hbase.client.TestAsyncProcess-with-HBASE-21775.txt
> The BufferedMutator doesn't ever refresh region location cache
> --
>
> Key: HBASE-21775
> URL: https://issues.apache.org/jira/browse/HBASE-21775
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Tommy Li
>Assignee: Tommy Li
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.1.3, 2.0.5, 1.3.4
>
> Attachments: HBASE-21775.master.001.patch,
> org.apache.hadoop.hbase.client.TestAsyncProcess-with-HBASE-21775.txt,
> org.apache.hadoop.hbase.client.TestAsyncProcess-without-HBASE-21775.txt
>
>
> {color:#22}I noticed in some of my writing jobs that the BufferedMutator
> would get stuck retrying writes against a dead server.{color}
> {code:java}
> 19/01/18 15:15:47 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:15:54 WARN [htable-pool3-t56] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=15/21, failureCount=1ops, last
> exception=org.apache.hadoop.hbase.DoNotRetryIOException: Operation rpcTimeout
> on ,17020,1547848193782, tracking started Fri Jan 18 14:55:37 PST
> 2019; NOT retrying, failed=1 -- final attempt!
> 19/01/18 15:15:54 ERROR [Executor task launch worker for task 0]
> IngestRawData.map(): [B@258bc2c7:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: Operation rpcTimeout: 1 time, servers with issues:
> ,17020,1547848193782
> {code}
>
> After the single remaining action permanently failed, it would resume
> progress only to get stuck again retrying against the same dead server:
> {code:java}
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:20 INFO [htable-pool3-t55] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=6/21, failureCount=1ops, last
> exception=java.net.ConnectException: Call to failed on connection
> exception:
> org.apache.hbase.thirdparty.io.netty.channel.ConnectTimeoutException:
> connection timed out: on ,17020,1547848193782, tracking
> started null, retrying after=20089ms, operationsToReplay=1
> {code}
>
> Only restarting the client process to generate a new BufferedMutator instance
> would fix the issue, at least until the next regionserver crash
> The logs I've pasted show the issue happening with a
> ConnectionTimeoutException, but we've also seen it with
> NotServingRegionException and some others
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21775) The BufferedMutator doesn't ever refresh region location cache
[
https://issues.apache.org/jira/browse/HBASE-21775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755296#comment-16755296
]
Tommy Li commented on HBASE-21775:
--
I'm looking at this. [~Apache9] can you paste a link to output of the failed
test? I'm not familiar with jenkins and am having trouble finding the failure
> The BufferedMutator doesn't ever refresh region location cache
> --
>
> Key: HBASE-21775
> URL: https://issues.apache.org/jira/browse/HBASE-21775
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Tommy Li
>Assignee: Tommy Li
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.1.3, 2.0.5, 1.3.4
>
> Attachments: HBASE-21775.master.001.patch
>
>
> {color:#22}I noticed in some of my writing jobs that the BufferedMutator
> would get stuck retrying writes against a dead server.{color}
> {code:java}
> 19/01/18 15:15:47 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:15:54 WARN [htable-pool3-t56] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=15/21, failureCount=1ops, last
> exception=org.apache.hadoop.hbase.DoNotRetryIOException: Operation rpcTimeout
> on ,17020,1547848193782, tracking started Fri Jan 18 14:55:37 PST
> 2019; NOT retrying, failed=1 -- final attempt!
> 19/01/18 15:15:54 ERROR [Executor task launch worker for task 0]
> IngestRawData.map(): [B@258bc2c7:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: Operation rpcTimeout: 1 time, servers with issues:
> ,17020,1547848193782
> {code}
>
> After the single remaining action permanently failed, it would resume
> progress only to get stuck again retrying against the same dead server:
> {code:java}
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:20 INFO [htable-pool3-t55] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=6/21, failureCount=1ops, last
> exception=java.net.ConnectException: Call to failed on connection
> exception:
> org.apache.hbase.thirdparty.io.netty.channel.ConnectTimeoutException:
> connection timed out: on ,17020,1547848193782, tracking
> started null, retrying after=20089ms, operationsToReplay=1
> {code}
>
> Only restarting the client process to generate a new BufferedMutator instance
> would fix the issue, at least until the next regionserver crash
> The logs I've pasted show the issue happening with a
> ConnectionTimeoutException, but we've also seen it with
> NotServingRegionException and some others
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21775) The BufferedMutator doesn't ever refresh region location cache
[
https://issues.apache.org/jira/browse/HBASE-21775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752856#comment-16752856
]
Tommy Li edited comment on HBASE-21775 at 1/26/19 12:37 AM:
[~stack] It definitely needs to go to branch-2. I haven't tested this on
version 1, but i took a brief look at the code and that condition is [the
same|https://github.com/apache/hbase/blob/branch-1.4/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java#L1259]
so yeah this can also go to branch-1
was (Author: tommyzli):
[~stack] It definitely needs to go to branch-2. I haven't tested this on
version 1, but i took a brief look at the code and that condition is [the
same|[https://github.com/apache/hbase/blob/branch-1.4/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java#L1259]|https://github.com/apache/hbase/blob/branch-1.4/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java#L1259],]
so yeah this can also go to branch-1
> The BufferedMutator doesn't ever refresh region location cache
> --
>
> Key: HBASE-21775
> URL: https://issues.apache.org/jira/browse/HBASE-21775
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Tommy Li
>Assignee: Tommy Li
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21775.master.001.patch
>
>
> {color:#22}I noticed in some of my writing jobs that the BufferedMutator
> would get stuck retrying writes against a dead server.{color}
> {code:java}
> 19/01/18 15:15:47 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:15:54 WARN [htable-pool3-t56] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=15/21, failureCount=1ops, last
> exception=org.apache.hadoop.hbase.DoNotRetryIOException: Operation rpcTimeout
> on ,17020,1547848193782, tracking started Fri Jan 18 14:55:37 PST
> 2019; NOT retrying, failed=1 -- final attempt!
> 19/01/18 15:15:54 ERROR [Executor task launch worker for task 0]
> IngestRawData.map(): [B@258bc2c7:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: Operation rpcTimeout: 1 time, servers with issues:
> ,17020,1547848193782
> {code}
>
> After the single remaining action permanently failed, it would resume
> progress only to get stuck again retrying against the same dead server:
> {code:java}
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:20 INFO [htable-pool3-t55] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=6/21, failureCount=1ops, last
> exception=java.net.ConnectException: Call to failed on connection
> exception:
> org.apache.hbase.thirdparty.io.netty.channel.ConnectTimeoutException:
> connection timed out: on ,17020,1547848193782, tracking
> started null, retrying after=20089ms, operationsToReplay=1
> {code}
>
> Only restarting the client process to generate a new BufferedMutator instance
> would fix the issue, at least until the next regionserver crash
> The logs I've pasted show the issue happening with a
> ConnectionTimeoutException, but we've also seen it with
> NotServingRegionException and some others
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21775) The BufferedMutator doesn't ever refresh region location cache
[
https://issues.apache.org/jira/browse/HBASE-21775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752856#comment-16752856
]
Tommy Li commented on HBASE-21775:
--
[~stack] It definitely needs to go to branch-2. I haven't tested this on
version 1, but i took a brief look at the code and that condition is [the
same|[https://github.com/apache/hbase/blob/branch-1.4/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java#L1259]|https://github.com/apache/hbase/blob/branch-1.4/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java#L1259],]
so yeah this can also go to branch-1
> The BufferedMutator doesn't ever refresh region location cache
> --
>
> Key: HBASE-21775
> URL: https://issues.apache.org/jira/browse/HBASE-21775
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Tommy Li
>Assignee: Tommy Li
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21775.master.001.patch
>
>
> {color:#22}I noticed in some of my writing jobs that the BufferedMutator
> would get stuck retrying writes against a dead server.{color}
> {code:java}
> 19/01/18 15:15:47 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:15:54 WARN [htable-pool3-t56] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=15/21, failureCount=1ops, last
> exception=org.apache.hadoop.hbase.DoNotRetryIOException: Operation rpcTimeout
> on ,17020,1547848193782, tracking started Fri Jan 18 14:55:37 PST
> 2019; NOT retrying, failed=1 -- final attempt!
> 19/01/18 15:15:54 ERROR [Executor task launch worker for task 0]
> IngestRawData.map(): [B@258bc2c7:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: Operation rpcTimeout: 1 time, servers with issues:
> ,17020,1547848193782
> {code}
>
> After the single remaining action permanently failed, it would resume
> progress only to get stuck again retrying against the same dead server:
> {code:java}
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:20 INFO [htable-pool3-t55] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=6/21, failureCount=1ops, last
> exception=java.net.ConnectException: Call to failed on connection
> exception:
> org.apache.hbase.thirdparty.io.netty.channel.ConnectTimeoutException:
> connection timed out: on ,17020,1547848193782, tracking
> started null, retrying after=20089ms, operationsToReplay=1
> {code}
>
> Only restarting the client process to generate a new BufferedMutator instance
> would fix the issue, at least until the next regionserver crash
> The logs I've pasted show the issue happening with a
> ConnectionTimeoutException, but we've also seen it with
> NotServingRegionException and some others
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21775) The BufferedMutator doesn't ever refresh region location cache
[
https://issues.apache.org/jira/browse/HBASE-21775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752658#comment-16752658
]
Tommy Li commented on HBASE-21775:
--
[~stack] yes - ran a quick test where I killed the cluster while the ingestion
process was running and confirmed that the buffered mutator picked up the new
region locations when it came back up.
> The BufferedMutator doesn't ever refresh region location cache
> --
>
> Key: HBASE-21775
> URL: https://issues.apache.org/jira/browse/HBASE-21775
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Tommy Li
>Assignee: Tommy Li
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21775.master.001.patch
>
>
> {color:#22}I noticed in some of my writing jobs that the BufferedMutator
> would get stuck retrying writes against a dead server.{color}
> {code:java}
> 19/01/18 15:15:47 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:15:54 WARN [htable-pool3-t56] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=15/21, failureCount=1ops, last
> exception=org.apache.hadoop.hbase.DoNotRetryIOException: Operation rpcTimeout
> on ,17020,1547848193782, tracking started Fri Jan 18 14:55:37 PST
> 2019; NOT retrying, failed=1 -- final attempt!
> 19/01/18 15:15:54 ERROR [Executor task launch worker for task 0]
> IngestRawData.map(): [B@258bc2c7:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: Operation rpcTimeout: 1 time, servers with issues:
> ,17020,1547848193782
> {code}
>
> After the single remaining action permanently failed, it would resume
> progress only to get stuck again retrying against the same dead server:
> {code:java}
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:20 INFO [htable-pool3-t55] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=6/21, failureCount=1ops, last
> exception=java.net.ConnectException: Call to failed on connection
> exception:
> org.apache.hbase.thirdparty.io.netty.channel.ConnectTimeoutException:
> connection timed out: on ,17020,1547848193782, tracking
> started null, retrying after=20089ms, operationsToReplay=1
> {code}
>
> Only restarting the client process to generate a new BufferedMutator instance
> would fix the issue, at least until the next regionserver crash
> The logs I've pasted show the issue happening with a
> ConnectionTimeoutException, but we've also seen it with
> NotServingRegionException and some others
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21775) The BufferedMutator doesn't ever refresh region location cache
[
https://issues.apache.org/jira/browse/HBASE-21775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752522#comment-16752522
]
Tommy Li commented on HBASE-21775:
--
[~stack] from what I can see, tableName shouldn't be null unless you manually
create a BufferedMutatorImpl instead of using
ConnectionFactory.createConnection().getBufferedMutator(). I not sure if the
bufferedmutator would work at all without a table name.
I'm running a build taken from master a few months ago, but I've seen the same
issue in the latest release.
> The BufferedMutator doesn't ever refresh region location cache
> --
>
> Key: HBASE-21775
> URL: https://issues.apache.org/jira/browse/HBASE-21775
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Tommy Li
>Assignee: Tommy Li
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21775.master.001.patch
>
>
> {color:#22}I noticed in some of my writing jobs that the BufferedMutator
> would get stuck retrying writes against a dead server.{color}
> {code:java}
> 19/01/18 15:15:47 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:15:54 WARN [htable-pool3-t56] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=15/21, failureCount=1ops, last
> exception=org.apache.hadoop.hbase.DoNotRetryIOException: Operation rpcTimeout
> on ,17020,1547848193782, tracking started Fri Jan 18 14:55:37 PST
> 2019; NOT retrying, failed=1 -- final attempt!
> 19/01/18 15:15:54 ERROR [Executor task launch worker for task 0]
> IngestRawData.map(): [B@258bc2c7:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: Operation rpcTimeout: 1 time, servers with issues:
> ,17020,1547848193782
> {code}
>
> After the single remaining action permanently failed, it would resume
> progress only to get stuck again retrying against the same dead server:
> {code:java}
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:20 INFO [htable-pool3-t55] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=6/21, failureCount=1ops, last
> exception=java.net.ConnectException: Call to failed on connection
> exception:
> org.apache.hbase.thirdparty.io.netty.channel.ConnectTimeoutException:
> connection timed out: on ,17020,1547848193782, tracking
> started null, retrying after=20089ms, operationsToReplay=1
> {code}
>
> Only restarting the client process to generate a new BufferedMutator instance
> would fix the issue, at least until the next regionserver crash
> The logs I've pasted show the issue happening with a
> ConnectionTimeoutException, but we've also seen it with
> NotServingRegionException and some others
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21775) The BufferedMutator doesn't ever refresh region location cache
[
https://issues.apache.org/jira/browse/HBASE-21775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tommy Li updated HBASE-21775:
-
Description:
{color:#22}I noticed in some of my writing jobs that the BufferedMutator
would get stuck retrying writes against a dead server.{color}
{code:java}
19/01/18 15:15:47 INFO [Executor task launch worker for task 0]
client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
dummy_table
19/01/18 15:15:54 WARN [htable-pool3-t56] client.AsyncRequestFutureImpl: id=2,
table=dummy_table, attempt=15/21, failureCount=1ops, last
exception=org.apache.hadoop.hbase.DoNotRetryIOException: Operation rpcTimeout
on ,17020,1547848193782, tracking started Fri Jan 18 14:55:37 PST 2019;
NOT retrying, failed=1 -- final attempt!
19/01/18 15:15:54 ERROR [Executor task launch worker for task 0]
IngestRawData.map(): [B@258bc2c7:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
action: Operation rpcTimeout: 1 time, servers with issues:
,17020,1547848193782
{code}
After the single remaining action permanently failed, it would resume progress
only to get stuck again retrying against the same dead server:
{code:java}
19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
dummy_table
19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
dummy_table
19/01/18 15:21:20 INFO [htable-pool3-t55] client.AsyncRequestFutureImpl: id=2,
table=dummy_table, attempt=6/21, failureCount=1ops, last
exception=java.net.ConnectException: Call to failed on connection
exception:
org.apache.hbase.thirdparty.io.netty.channel.ConnectTimeoutException:
connection timed out: on ,17020,1547848193782, tracking
started null, retrying after=20089ms, operationsToReplay=1
{code}
Only restarting the client process to generate a new BufferedMutator instance
would fix the issue, at least until the next regionserver crash
The logs I've pasted show the issue happening with a
ConnectionTimeoutException, but we've also seen it with
NotServingRegionException and some others
was:
{color:#22}I noticed in some of my writing jobs that the BufferedMutator
would get stuck retrying writes against a dead server.{color}
{code:java}
19/01/18 15:15:47 INFO [Executor task launch worker for task 0]
client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
dummy_table
19/01/18 15:15:54 WARN [htable-pool3-t56] client.AsyncRequestFutureImpl: id=2,
table=dummy_table, attempt=15/21, failureCount=1ops, last
exception=org.apache.hadoop.hbase.DoNotRetryIOException: Operation rpcTimeout
on ,17020,1547848193782, tracking started Fri Jan 18 14:55:37 PST 2019;
NOT retrying, failed=1 -- final attempt!
19/01/18 15:15:54 ERROR [Executor task launch worker for task 0]
IngestRawData.map(): [B@258bc2c7:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
action: Operation rpcTimeout: 1 time, servers with issues:
,17020,1547848193782
{code}
After the single remaining action permanently failed, it would resume progress
only to get stuck again retrying against the same dead server:
{code:java}
19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
dummy_table
19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
dummy_table
19/01/18 15:21:20 INFO [htable-pool3-t55] client.AsyncRequestFutureImpl: id=2,
table=dummy_table, attempt=6/21, failureCount=1ops, last
exception=java.net.ConnectException: Call to failed on connection
exception:
org.apache.hbase.thirdparty.io.netty.channel.ConnectTimeoutException:
connection timed out: on ,17020,1547848193782, tracking
started null, retrying after=20089ms, operationsToReplay=1
{code}
Only restarting the client process to generate a new BufferedMutator instance
would fix the issue, at least until the next regionserver crash
> The BufferedMutator doesn't ever refresh region location cache
> --
>
> Key: HBASE-21775
> URL: https://issues.apache.org/jira/browse/HBASE-21775
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Tommy Li
>Assignee: Tommy Li
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21775.master.001.patch
>
>
> {color:#22}I noticed in some of my writing jobs that the BufferedMutator
> would get stuck retrying writes against a dead server.{color}
> {code:java}
> 19/01/18 15:15:47 INFO [Executor task launch worker for task 0]
>
[jira] [Updated] (HBASE-21775) The BufferedMutator doesn't ever refresh region location cache
[
https://issues.apache.org/jira/browse/HBASE-21775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tommy Li updated HBASE-21775:
-
Attachment: HBASE-21775.master.001.patch
> The BufferedMutator doesn't ever refresh region location cache
> --
>
> Key: HBASE-21775
> URL: https://issues.apache.org/jira/browse/HBASE-21775
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Tommy Li
>Assignee: Tommy Li
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21775.master.001.patch
>
>
> {color:#22}I noticed in some of my writing jobs that the BufferedMutator
> would get stuck retrying writes against a dead server.{color}
> {code:java}
> 19/01/18 15:15:47 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:15:54 WARN [htable-pool3-t56] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=15/21, failureCount=1ops, last
> exception=org.apache.hadoop.hbase.DoNotRetryIOException: Operation rpcTimeout
> on ,17020,1547848193782, tracking started Fri Jan 18 14:55:37 PST
> 2019; NOT retrying, failed=1 -- final attempt!
> 19/01/18 15:15:54 ERROR [Executor task launch worker for task 0]
> IngestRawData.map(): [B@258bc2c7:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: Operation rpcTimeout: 1 time, servers with issues:
> ,17020,1547848193782
> {code}
>
> After the single remaining action permanently failed, it would resume
> progress only to get stuck again retrying against the same dead server:
> {code:java}
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:20 INFO [htable-pool3-t55] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=6/21, failureCount=1ops, last
> exception=java.net.ConnectException: Call to failed on connection
> exception:
> org.apache.hbase.thirdparty.io.netty.channel.ConnectTimeoutException:
> connection timed out: on ,17020,1547848193782, tracking
> started null, retrying after=20089ms, operationsToReplay=1
> {code}
>
> Only restarting the client process to generate a new BufferedMutator instance
> would fix the issue, at least until the next regionserver crash
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21775) The BufferedMutator doesn't ever refresh region location cache
[
https://issues.apache.org/jira/browse/HBASE-21775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tommy Li updated HBASE-21775:
-
Status: Patch Available (was: Open)
> The BufferedMutator doesn't ever refresh region location cache
> --
>
> Key: HBASE-21775
> URL: https://issues.apache.org/jira/browse/HBASE-21775
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Tommy Li
>Assignee: Tommy Li
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21775.master.001.patch
>
>
> {color:#22}I noticed in some of my writing jobs that the BufferedMutator
> would get stuck retrying writes against a dead server.{color}
> {code:java}
> 19/01/18 15:15:47 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:15:54 WARN [htable-pool3-t56] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=15/21, failureCount=1ops, last
> exception=org.apache.hadoop.hbase.DoNotRetryIOException: Operation rpcTimeout
> on ,17020,1547848193782, tracking started Fri Jan 18 14:55:37 PST
> 2019; NOT retrying, failed=1 -- final attempt!
> 19/01/18 15:15:54 ERROR [Executor task launch worker for task 0]
> IngestRawData.map(): [B@258bc2c7:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: Operation rpcTimeout: 1 time, servers with issues:
> ,17020,1547848193782
> {code}
>
> After the single remaining action permanently failed, it would resume
> progress only to get stuck again retrying against the same dead server:
> {code:java}
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
> client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
> dummy_table
> 19/01/18 15:21:20 INFO [htable-pool3-t55] client.AsyncRequestFutureImpl:
> id=2, table=dummy_table, attempt=6/21, failureCount=1ops, last
> exception=java.net.ConnectException: Call to failed on connection
> exception:
> org.apache.hbase.thirdparty.io.netty.channel.ConnectTimeoutException:
> connection timed out: on ,17020,1547848193782, tracking
> started null, retrying after=20089ms, operationsToReplay=1
> {code}
>
> Only restarting the client process to generate a new BufferedMutator instance
> would fix the issue, at least until the next regionserver crash
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21775) The BufferedMutator doesn't ever refresh region location cache
Tommy Li created HBASE-21775:
Summary: The BufferedMutator doesn't ever refresh region location
cache
Key: HBASE-21775
URL: https://issues.apache.org/jira/browse/HBASE-21775
Project: HBase
Issue Type: Bug
Components: Client
Reporter: Tommy Li
Assignee: Tommy Li
Fix For: 3.0.0
{color:#22}I noticed in some of my writing jobs that the BufferedMutator
would get stuck retrying writes against a dead server.{color}
{code:java}
19/01/18 15:15:47 INFO [Executor task launch worker for task 0]
client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
dummy_table
19/01/18 15:15:54 WARN [htable-pool3-t56] client.AsyncRequestFutureImpl: id=2,
table=dummy_table, attempt=15/21, failureCount=1ops, last
exception=org.apache.hadoop.hbase.DoNotRetryIOException: Operation rpcTimeout
on ,17020,1547848193782, tracking started Fri Jan 18 14:55:37 PST 2019;
NOT retrying, failed=1 -- final attempt!
19/01/18 15:15:54 ERROR [Executor task launch worker for task 0]
IngestRawData.map(): [B@258bc2c7:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
action: Operation rpcTimeout: 1 time, servers with issues:
,17020,1547848193782
{code}
After the single remaining action permanently failed, it would resume progress
only to get stuck again retrying against the same dead server:
{code:java}
19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
dummy_table
19/01/18 15:21:18 INFO [Executor task launch worker for task 0]
client.AsyncRequestFutureImpl: #2, waiting for 1 actions to finish on table:
dummy_table
19/01/18 15:21:20 INFO [htable-pool3-t55] client.AsyncRequestFutureImpl: id=2,
table=dummy_table, attempt=6/21, failureCount=1ops, last
exception=java.net.ConnectException: Call to failed on connection
exception:
org.apache.hbase.thirdparty.io.netty.channel.ConnectTimeoutException:
connection timed out: on ,17020,1547848193782, tracking
started null, retrying after=20089ms, operationsToReplay=1
{code}
Only restarting the client process to generate a new BufferedMutator instance
would fix the issue, at least until the next regionserver crash
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21720) metric to measure how actions are distributed to servers within a MultiAction
[ https://issues.apache.org/jira/browse/HBASE-21720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommy Li updated HBASE-21720: - Attachment: HBASE-21720.master.006.patch > metric to measure how actions are distributed to servers within a MultiAction > - > > Key: HBASE-21720 > URL: https://issues.apache.org/jira/browse/HBASE-21720 > Project: HBase > Issue Type: Improvement > Components: Client, metrics, monitoring >Reporter: Tommy Li >Assignee: Tommy Li >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-21720.master.001.patch, > HBASE-21720.master.002.patch, HBASE-21720.master.003.patch, > HBASE-21720.master.004.patch, HBASE-21720.master.005.patch, > HBASE-21720.master.006.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21720) metric to measure how actions are distributed to servers within a MultiAction
[ https://issues.apache.org/jira/browse/HBASE-21720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749256#comment-16749256 ] Tommy Li commented on HBASE-21720: -- The checkstyle is complaining about some things not directly introduced by my patch. I've fixed the largest issue - the indentation of the large case statement in MetricsConnection#updateRpc, but I can undo that if anyone feels it's polluting the meat of the patch. > metric to measure how actions are distributed to servers within a MultiAction > - > > Key: HBASE-21720 > URL: https://issues.apache.org/jira/browse/HBASE-21720 > Project: HBase > Issue Type: Improvement > Components: Client, metrics, monitoring >Reporter: Tommy Li >Assignee: Tommy Li >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-21720.master.001.patch, > HBASE-21720.master.002.patch, HBASE-21720.master.003.patch, > HBASE-21720.master.004.patch, HBASE-21720.master.005.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21720) metric to measure how actions are distributed to servers within a MultiAction
[ https://issues.apache.org/jira/browse/HBASE-21720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommy Li updated HBASE-21720: - Attachment: HBASE-21720.master.005.patch > metric to measure how actions are distributed to servers within a MultiAction > - > > Key: HBASE-21720 > URL: https://issues.apache.org/jira/browse/HBASE-21720 > Project: HBase > Issue Type: Improvement > Components: Client, metrics, monitoring >Reporter: Tommy Li >Assignee: Tommy Li >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-21720.master.001.patch, > HBASE-21720.master.002.patch, HBASE-21720.master.003.patch, > HBASE-21720.master.004.patch, HBASE-21720.master.005.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21720) metric to measure how actions are distributed to servers within a MultiAction
[ https://issues.apache.org/jira/browse/HBASE-21720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommy Li updated HBASE-21720: - Attachment: HBASE-21720.master.004.patch > metric to measure how actions are distributed to servers within a MultiAction > - > > Key: HBASE-21720 > URL: https://issues.apache.org/jira/browse/HBASE-21720 > Project: HBase > Issue Type: Improvement > Components: Client, metrics, monitoring >Reporter: Tommy Li >Assignee: Tommy Li >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-21720.master.001.patch, > HBASE-21720.master.002.patch, HBASE-21720.master.003.patch, > HBASE-21720.master.004.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21720) metric to measure how actions are distributed to servers within a MultiAction
[ https://issues.apache.org/jira/browse/HBASE-21720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommy Li updated HBASE-21720: - Attachment: HBASE-21720.master.003.patch > metric to measure how actions are distributed to servers within a MultiAction > - > > Key: HBASE-21720 > URL: https://issues.apache.org/jira/browse/HBASE-21720 > Project: HBase > Issue Type: Improvement > Components: Client, metrics, monitoring >Reporter: Tommy Li >Assignee: Tommy Li >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-21720.master.001.patch, > HBASE-21720.master.002.patch, HBASE-21720.master.003.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21712) Make submit-patch.py python3 compatible
[ https://issues.apache.org/jira/browse/HBASE-21712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743395#comment-16743395 ] Tommy Li commented on HBASE-21712: -- [~psomogyi] I've attached HBASE-21722.master.001.patch > Make submit-patch.py python3 compatible > --- > > Key: HBASE-21712 > URL: https://issues.apache.org/jira/browse/HBASE-21712 > Project: HBase > Issue Type: Improvement > Components: tooling >Reporter: Tommy Li >Assignee: Tommy Li >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-21712.master.001.patch, > HBASE-21712.master.002.patch, HBASE-21722.master.001.patch.txt > > > Attached patch was submitted with `python3 dev-support/submit-patch.py -b > master -srb -jid HBASE-21712` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21712) Make submit-patch.py python3 compatible
[ https://issues.apache.org/jira/browse/HBASE-21712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommy Li updated HBASE-21712: - Attachment: HBASE-21722.master.001.patch.txt Status: Patch Available (was: Reopened) > Make submit-patch.py python3 compatible > --- > > Key: HBASE-21712 > URL: https://issues.apache.org/jira/browse/HBASE-21712 > Project: HBase > Issue Type: Improvement > Components: tooling >Reporter: Tommy Li >Assignee: Tommy Li >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-21712.master.001.patch, > HBASE-21712.master.002.patch, HBASE-21722.master.001.patch.txt > > > Attached patch was submitted with `python3 dev-support/submit-patch.py -b > master -srb -jid HBASE-21712` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21720) metric to measure how actions are distributed to servers within a MultiAction
[ https://issues.apache.org/jira/browse/HBASE-21720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommy Li updated HBASE-21720: - Attachment: HBASE-21720.master.002.patch > metric to measure how actions are distributed to servers within a MultiAction > - > > Key: HBASE-21720 > URL: https://issues.apache.org/jira/browse/HBASE-21720 > Project: HBase > Issue Type: Improvement > Components: Client, metrics, monitoring >Reporter: Tommy Li >Assignee: Tommy Li >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-21720.master.001.patch, > HBASE-21720.master.002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21712) Make submit-patch.py python3 compatible
[ https://issues.apache.org/jira/browse/HBASE-21712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16742806#comment-16742806 ] Tommy Li commented on HBASE-21712: -- My bad, forgot to update the python requirements file in this patch. fixed in https://issues.apache.org/jira/browse/HBASE-21722 > Make submit-patch.py python3 compatible > --- > > Key: HBASE-21712 > URL: https://issues.apache.org/jira/browse/HBASE-21712 > Project: HBase > Issue Type: Improvement > Components: tooling >Reporter: Tommy Li >Assignee: Tommy Li >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-21712.master.001.patch, > HBASE-21712.master.002.patch > > > Attached patch was submitted with `python3 dev-support/submit-patch.py -b > master -srb -jid HBASE-21712` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21722) Add missing python requirement to dev-support
[ https://issues.apache.org/jira/browse/HBASE-21722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommy Li updated HBASE-21722: - Attachment: HBASE-21722.master.001.patch Status: Patch Available (was: Open) > Add missing python requirement to dev-support > - > > Key: HBASE-21722 > URL: https://issues.apache.org/jira/browse/HBASE-21722 > Project: HBase > Issue Type: Bug >Reporter: Tommy Li >Assignee: Tommy Li >Priority: Minor > Attachments: HBASE-21722.master.001.patch > > > submit-patch.py now requries the future module -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21722) Add missing python requirement to dev-support
Tommy Li created HBASE-21722: Summary: Add missing python requirement to dev-support Key: HBASE-21722 URL: https://issues.apache.org/jira/browse/HBASE-21722 Project: HBase Issue Type: Bug Reporter: Tommy Li Assignee: Tommy Li submit-patch.py now requries the future module -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21712) Make submit-patch.py python3 compatible
[ https://issues.apache.org/jira/browse/HBASE-21712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16742702#comment-16742702 ] Tommy Li commented on HBASE-21712: -- [~jojochuang] what version of python are you using? The builtins module should exist in all supported versions of python > Make submit-patch.py python3 compatible > --- > > Key: HBASE-21712 > URL: https://issues.apache.org/jira/browse/HBASE-21712 > Project: HBase > Issue Type: Improvement > Components: tooling >Reporter: Tommy Li >Assignee: Tommy Li >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-21712.master.001.patch, > HBASE-21712.master.002.patch > > > Attached patch was submitted with `python3 dev-support/submit-patch.py -b > master -srb -jid HBASE-21712` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21720) metric to measure how actions are distributed to servers within a MultiAction
Tommy Li created HBASE-21720: Summary: metric to measure how actions are distributed to servers within a MultiAction Key: HBASE-21720 URL: https://issues.apache.org/jira/browse/HBASE-21720 Project: HBase Issue Type: Improvement Components: Client, metrics, monitoring Reporter: Tommy Li Assignee: Tommy Li Fix For: 3.0.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21720) metric to measure how actions are distributed to servers within a MultiAction
[ https://issues.apache.org/jira/browse/HBASE-21720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommy Li updated HBASE-21720: - Status: Patch Available (was: Open) > metric to measure how actions are distributed to servers within a MultiAction > - > > Key: HBASE-21720 > URL: https://issues.apache.org/jira/browse/HBASE-21720 > Project: HBase > Issue Type: Improvement > Components: Client, metrics, monitoring >Reporter: Tommy Li >Assignee: Tommy Li >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-21720.master.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21720) metric to measure how actions are distributed to servers within a MultiAction
[ https://issues.apache.org/jira/browse/HBASE-21720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommy Li updated HBASE-21720: - Attachment: HBASE-21720.master.001.patch > metric to measure how actions are distributed to servers within a MultiAction > - > > Key: HBASE-21720 > URL: https://issues.apache.org/jira/browse/HBASE-21720 > Project: HBase > Issue Type: Improvement > Components: Client, metrics, monitoring >Reporter: Tommy Li >Assignee: Tommy Li >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-21720.master.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21712) Make submit-patch.py python3 compatible
[ https://issues.apache.org/jira/browse/HBASE-21712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommy Li updated HBASE-21712: - Attachment: HBASE-21712.master.002.patch > Make submit-patch.py python3 compatible > --- > > Key: HBASE-21712 > URL: https://issues.apache.org/jira/browse/HBASE-21712 > Project: HBase > Issue Type: Improvement > Components: tooling >Reporter: Tommy Li >Assignee: Tommy Li >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-21712.master.001.patch, > HBASE-21712.master.002.patch > > > Attached patch was submitted with `python3 dev-support/submit-patch.py -b > master -srb -jid HBASE-21712` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21712) Make submit-patch.py python3 compatible
[ https://issues.apache.org/jira/browse/HBASE-21712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommy Li updated HBASE-21712: - Status: Patch Available (was: Open) > Make submit-patch.py python3 compatible > --- > > Key: HBASE-21712 > URL: https://issues.apache.org/jira/browse/HBASE-21712 > Project: HBase > Issue Type: Improvement > Components: tooling >Reporter: Tommy Li >Assignee: Tommy Li >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-21712.master.001.patch > > > Attached patch was submitted with `python3 dev-support/submit-patch.py -b > master -srb -jid HBASE-21712` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21712) Make submit-patch.py python3 compatible
[ https://issues.apache.org/jira/browse/HBASE-21712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommy Li updated HBASE-21712: - Attachment: HBASE-21712.master.001.patch > Make submit-patch.py python3 compatible > --- > > Key: HBASE-21712 > URL: https://issues.apache.org/jira/browse/HBASE-21712 > Project: HBase > Issue Type: Improvement > Components: tooling >Reporter: Tommy Li >Assignee: Tommy Li >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-21712.master.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21712) Make submit-patch.py python3 compatible
[ https://issues.apache.org/jira/browse/HBASE-21712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommy Li updated HBASE-21712: - Description: Attached patch was submitted with `python3 dev-support/submit-patch.py -b master -srb -jid HBASE-21712` > Make submit-patch.py python3 compatible > --- > > Key: HBASE-21712 > URL: https://issues.apache.org/jira/browse/HBASE-21712 > Project: HBase > Issue Type: Improvement > Components: tooling >Reporter: Tommy Li >Assignee: Tommy Li >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-21712.master.001.patch > > > Attached patch was submitted with `python3 dev-support/submit-patch.py -b > master -srb -jid HBASE-21712` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21712) Make submit-patch.py python3 compatible
Tommy Li created HBASE-21712: Summary: Make submit-patch.py python3 compatible Key: HBASE-21712 URL: https://issues.apache.org/jira/browse/HBASE-21712 Project: HBase Issue Type: Improvement Components: tooling Reporter: Tommy Li Assignee: Tommy Li Fix For: 3.0.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21414) StoreFileSize growth rate metric
[ https://issues.apache.org/jira/browse/HBASE-21414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommy Li updated HBASE-21414: - Attachment: HBASE-21414.master.003.patch > StoreFileSize growth rate metric > > > Key: HBASE-21414 > URL: https://issues.apache.org/jira/browse/HBASE-21414 > Project: HBase > Issue Type: Improvement > Components: metrics, monitoring >Reporter: Tommy Li >Assignee: Tommy Li >Priority: Minor > Attachments: HBASE-21414.master.001.patch, > HBASE-21414.master.002.patch, HBASE-21414.master.003.patch > > > A metric on the growth rate of storefile sizes would be nice to have as a way > of monitoring traffic patterns. I know you can get the same insight from > graphing the delta on the storeFileSize metric, but not all metrics > visualization tools support that -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21414) StoreFileSize growth rate metric
[ https://issues.apache.org/jira/browse/HBASE-21414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691129#comment-16691129 ] Tommy Li commented on HBASE-21414: -- Thanks [~apurtell] - I've addressed the nit in the second patch > StoreFileSize growth rate metric > > > Key: HBASE-21414 > URL: https://issues.apache.org/jira/browse/HBASE-21414 > Project: HBase > Issue Type: Improvement > Components: metrics, monitoring >Reporter: Tommy Li >Priority: Minor > Attachments: HBASE-21414.master.001.patch, > HBASE-21414.master.002.patch > > > A metric on the growth rate of storefile sizes would be nice to have as a way > of monitoring traffic patterns. I know you can get the same insight from > graphing the delta on the storeFileSize metric, but not all metrics > visualization tools support that -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21414) StoreFileSize growth rate metric
[ https://issues.apache.org/jira/browse/HBASE-21414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommy Li updated HBASE-21414: - Attachment: HBASE-21414.master.002.patch > StoreFileSize growth rate metric > > > Key: HBASE-21414 > URL: https://issues.apache.org/jira/browse/HBASE-21414 > Project: HBase > Issue Type: Improvement > Components: metrics, monitoring >Reporter: Tommy Li >Priority: Minor > Attachments: HBASE-21414.master.001.patch, > HBASE-21414.master.002.patch > > > A metric on the growth rate of storefile sizes would be nice to have as a way > of monitoring traffic patterns. I know you can get the same insight from > graphing the delta on the storeFileSize metric, but not all metrics > visualization tools support that -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21414) StoreFileSize growth rate metric
[ https://issues.apache.org/jira/browse/HBASE-21414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommy Li updated HBASE-21414: - Attachment: (was: HBASE-21414.master.002.patch) > StoreFileSize growth rate metric > > > Key: HBASE-21414 > URL: https://issues.apache.org/jira/browse/HBASE-21414 > Project: HBase > Issue Type: Improvement > Components: metrics, monitoring >Reporter: Tommy Li >Priority: Minor > Attachments: HBASE-21414.master.001.patch > > > A metric on the growth rate of storefile sizes would be nice to have as a way > of monitoring traffic patterns. I know you can get the same insight from > graphing the delta on the storeFileSize metric, but not all metrics > visualization tools support that -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21414) StoreFileSize growth rate metric
[ https://issues.apache.org/jira/browse/HBASE-21414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommy Li updated HBASE-21414: - Attachment: HBASE-21414.master.002.patch > StoreFileSize growth rate metric > > > Key: HBASE-21414 > URL: https://issues.apache.org/jira/browse/HBASE-21414 > Project: HBase > Issue Type: Improvement > Components: metrics, monitoring >Reporter: Tommy Li >Priority: Minor > Attachments: HBASE-21414.master.001.patch, > HBASE-21414.master.002.patch > > > A metric on the growth rate of storefile sizes would be nice to have as a way > of monitoring traffic patterns. I know you can get the same insight from > graphing the delta on the storeFileSize metric, but not all metrics > visualization tools support that -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21414) StoreFileSize growth rate metric
[ https://issues.apache.org/jira/browse/HBASE-21414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommy Li updated HBASE-21414: - Attachment: HBASE-21414.master.001.patch > StoreFileSize growth rate metric > > > Key: HBASE-21414 > URL: https://issues.apache.org/jira/browse/HBASE-21414 > Project: HBase > Issue Type: Improvement > Components: metrics, monitoring >Reporter: Tommy Li >Priority: Minor > Attachments: HBASE-21414.master.001.patch > > > A metric on the growth rate of storefile sizes would be nice to have as a way > of monitoring traffic patterns. I know you can get the same insight from > graphing the delta on the storeFileSize metric, but not all metrics > visualization tools support that -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21414) StoreFileSize growth rate metric
Tommy Li created HBASE-21414: Summary: StoreFileSize growth rate metric Key: HBASE-21414 URL: https://issues.apache.org/jira/browse/HBASE-21414 Project: HBase Issue Type: Improvement Components: metrics, monitoring Reporter: Tommy Li A metric on the growth rate of storefile sizes would be nice to have as a way of monitoring traffic patterns. I know you can get the same insight from graphing the delta on the storeFileSize metric, but not all metrics visualization tools support that -- This message was sent by Atlassian JIRA (v7.6.3#76005)
