[jira] [Commented] (HBASE-25524) Region Servers should give some indication that short-circuit reads are configured properly and working
[ https://issues.apache.org/jira/browse/HBASE-25524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17356565#comment-17356565 ] Sean Busbey commented on HBASE-25524: - We should be able to use the read from FSDataInputStreamWrapper to verify that the request wasn't served locally at all. We can use that to either do a retry or make a WARN about the canary being invalid. > Region Servers should give some indication that short-circuit reads are > configured properly and working > --- > > Key: HBASE-25524 > URL: https://issues.apache.org/jira/browse/HBASE-25524 > Project: HBase > Issue Type: Improvement > Components: Performance, regionserver, Usability >Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.4.0 >Reporter: Nick Dimiduk >Assignee: Rahul Kumar >Priority: Major > > Going through the exercise of confirming short-circuit reads turns out to be > quite a chore. 2.3.0 added a new metric, but metrics only get tickled when > there's traffic that happens to use the feature. This is not very convenient > for an operator who has taken a cluster out of service for maintenance. > It would be nice if the rs process could perform some test and emitted some > confirmation of the status of this otherwise transparent feature. After > confirmation, it could light up a boolean metric value and/or add mention of > it to the info page. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25524) Region Servers should give some indication that short-circuit reads are configured properly and working
[ https://issues.apache.org/jira/browse/HBASE-25524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17356564#comment-17356564 ] Sean Busbey commented on HBASE-25524: - ACDC requires the following justification for disclosure: Who requested you be disclosed The reason you were given to request disclosure > Region Servers should give some indication that short-circuit reads are > configured properly and working > --- > > Key: HBASE-25524 > URL: https://issues.apache.org/jira/browse/HBASE-25524 > Project: HBase > Issue Type: Improvement > Components: Performance, regionserver, Usability >Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.4.0 >Reporter: Nick Dimiduk >Assignee: Rahul Kumar >Priority: Major > > Going through the exercise of confirming short-circuit reads turns out to be > quite a chore. 2.3.0 added a new metric, but metrics only get tickled when > there's traffic that happens to use the feature. This is not very convenient > for an operator who has taken a cluster out of service for maintenance. > It would be nice if the rs process could perform some test and emitted some > confirmation of the status of this otherwise transparent feature. After > confirmation, it could light up a boolean metric value and/or add mention of > it to the info page. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25524) Region Servers should give some indication that short-circuit reads are configured properly and working
[ https://issues.apache.org/jira/browse/HBASE-25524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17356345#comment-17356345 ] Rahul Kumar commented on HBASE-25524: - Ping [~busbey] on the above concern. > Region Servers should give some indication that short-circuit reads are > configured properly and working > --- > > Key: HBASE-25524 > URL: https://issues.apache.org/jira/browse/HBASE-25524 > Project: HBase > Issue Type: Improvement > Components: Performance, regionserver, Usability >Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.4.0 >Reporter: Nick Dimiduk >Assignee: Rahul Kumar >Priority: Major > > Going through the exercise of confirming short-circuit reads turns out to be > quite a chore. 2.3.0 added a new metric, but metrics only get tickled when > there's traffic that happens to use the feature. This is not very convenient > for an operator who has taken a cluster out of service for maintenance. > It would be nice if the rs process could perform some test and emitted some > confirmation of the status of this otherwise transparent feature. After > confirmation, it could light up a boolean metric value and/or add mention of > it to the info page. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25524) Region Servers should give some indication that short-circuit reads are configured properly and working
[ https://issues.apache.org/jira/browse/HBASE-25524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17349651#comment-17349651 ] Rahul Kumar commented on HBASE-25524: - > but we can use the FSDataInputStreamWrapper we maintain to read the file and >then check if the read happened via SCR / zero copy [Sean Busbey|applewebdata://172D0324-EBDD-4FCD-9F17-B39723DE1DCC/jira/secure/ViewProfile.jspa?name=busbey] If the canary file isn't written/persistent local to the system, wouldn't that fail our purpose of ensuring SCR is working ? Sorry, looks like I am missing some context here. > Region Servers should give some indication that short-circuit reads are > configured properly and working > --- > > Key: HBASE-25524 > URL: https://issues.apache.org/jira/browse/HBASE-25524 > Project: HBase > Issue Type: Improvement > Components: Performance, regionserver, Usability >Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.4.0 >Reporter: Nick Dimiduk >Assignee: Rahul Kumar >Priority: Major > > Going through the exercise of confirming short-circuit reads turns out to be > quite a chore. 2.3.0 added a new metric, but metrics only get tickled when > there's traffic that happens to use the feature. This is not very convenient > for an operator who has taken a cluster out of service for maintenance. > It would be nice if the rs process could perform some test and emitted some > confirmation of the status of this otherwise transparent feature. After > confirmation, it could light up a boolean metric value and/or add mention of > it to the info page. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25524) Region Servers should give some indication that short-circuit reads are configured properly and working
[ https://issues.apache.org/jira/browse/HBASE-25524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17335785#comment-17335785 ] Sean Busbey commented on HBASE-25524: - If we want the ability to confirm that a RS _can_ do SCR independent of the specific regions it is hosting then I think we need to use a canary file. I think #2 above is the only practical way to do that. Presumably we only need to do this once per RS process. I don't think it matters that later readings of hfiles might not have a local block to use for a SCR; that notion is captured in locality and scr metrics. After writing the canary file I don't see a straight forward way to check that HDFS wrote the first block locally, but we can use the FSDataInputStreamWrapper we maintain to read the file and then check if the read happened via SCR / zero copy > Region Servers should give some indication that short-circuit reads are > configured properly and working > --- > > Key: HBASE-25524 > URL: https://issues.apache.org/jira/browse/HBASE-25524 > Project: HBase > Issue Type: Improvement > Components: Performance, regionserver, Usability >Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.4.0 >Reporter: Nick Dimiduk >Assignee: Rahul Kumar >Priority: Major > > Going through the exercise of confirming short-circuit reads turns out to be > quite a chore. 2.3.0 added a new metric, but metrics only get tickled when > there's traffic that happens to use the feature. This is not very convenient > for an operator who has taken a cluster out of service for maintenance. > It would be nice if the rs process could perform some test and emitted some > confirmation of the status of this otherwise transparent feature. After > confirmation, it could light up a boolean metric value and/or add mention of > it to the info page. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25524) Region Servers should give some indication that short-circuit reads are configured properly and working
[ https://issues.apache.org/jira/browse/HBASE-25524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17335755#comment-17335755 ] Nick Dimiduk commented on HBASE-25524: -- Writing and reading back a temporary file seems like a good strategy. Do we have any introspection into the stream at all, to know if it's using the local file handle? > Region Servers should give some indication that short-circuit reads are > configured properly and working > --- > > Key: HBASE-25524 > URL: https://issues.apache.org/jira/browse/HBASE-25524 > Project: HBase > Issue Type: Improvement > Components: Performance, regionserver, Usability >Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.4.0 >Reporter: Nick Dimiduk >Assignee: Rahul Kumar >Priority: Major > > Going through the exercise of confirming short-circuit reads turns out to be > quite a chore. 2.3.0 added a new metric, but metrics only get tickled when > there's traffic that happens to use the feature. This is not very convenient > for an operator who has taken a cluster out of service for maintenance. > It would be nice if the rs process could perform some test and emitted some > confirmation of the status of this otherwise transparent feature. After > confirmation, it could light up a boolean metric value and/or add mention of > it to the info page. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25524) Region Servers should give some indication that short-circuit reads are configured properly and working
[ https://issues.apache.org/jira/browse/HBASE-25524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329133#comment-17329133 ] Rahul Kumar commented on HBASE-25524: - Hey [~ndimiduk] . Thinking over the approach to validate SCR (short-circuit read). Couple of options that came into my mind, but facing some limitation with those. 1. Do a test read from RS and check if the read was SCR ? > Is there a way to always read a sample data which are always local to the > host ? 2. Try writing data locally and then read the same data to confirm SCR ? > Even if we write a test data to local, we can't be fully sure it will be > there in local at the time of reading. Do you see any better way that I am not able to see here ? Thanks > Region Servers should give some indication that short-circuit reads are > configured properly and working > --- > > Key: HBASE-25524 > URL: https://issues.apache.org/jira/browse/HBASE-25524 > Project: HBase > Issue Type: Improvement > Components: Performance, regionserver, Usability >Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.4.0 >Reporter: Nick Dimiduk >Assignee: Rahul Kumar >Priority: Major > > Going through the exercise of confirming short-circuit reads turns out to be > quite a chore. 2.3.0 added a new metric, but metrics only get tickled when > there's traffic that happens to use the feature. This is not very convenient > for an operator who has taken a cluster out of service for maintenance. > It would be nice if the rs process could perform some test and emitted some > confirmation of the status of this otherwise transparent feature. After > confirmation, it could light up a boolean metric value and/or add mention of > it to the info page. -- This message was sent by Atlassian Jira (v8.3.4#803005)
