[jira] [Comment Edited] (HBASE-18846) Accommodate the hbase-indexer/lily/SEP consumer deploy-type
[ https://issues.apache.org/jira/browse/HBASE-18846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214095#comment-16214095 ] stack edited comment on HBASE-18846 at 10/22/17 4:44 AM: - bq. There's a lot of unrelated whitespace/formatting changes in the patch. I'll try to go through sometime and read it with a tool that can ignore those. Sorry about that. Let me try and remove some. This stuff needs a refactor badly. I can't help myself. bq. I guess we still check if the handler is enabled, but it looks less clean than the other options we're exposing. Replication startup is complicated (I've yet to figure it out). Default in hbase is to start Replication. In the hollowed-out HRegionServer, we need Replication so we put up a Sink to receive Replication. On enable/disable of Replication, I'd like to do that in another issue. We have a million services/chores/managers and they all work different. Ideally, they'd all work the same and the flag to stop/start it would be same pattern no matter what the Service; but that is TODO I'd say. Idea behind this issue is to get the hbase-indexer folks something they can work with; my guess is they will come back with other stuff they'd like to be handled in subsequent issues before beta-1. Long-term, this masterless HRegionServer looks like it'll be useful testing HRegionServer in Isolation. bq. On the health check, can we get into a false positive state where a thread is null but shouldn't be? Is it better to check the configuration source of truth there instead? Perhaps. I'd like some more experience with hollow-RegionServer deploy first. Can fix in follow-on. bq. In the TestStandaloneRegionServer, let's put some of the cleanup (like the join) in either a finally block or a @After method so that we don't pollute other tests if we fail. Done (I tried to add more tests but short-circuiting RPC'ing messes up ability to test stuff like whether the service enable/disable are having an effect.). was (Author: stack): bq. There's a lot of unrelated whitespace/formatting changes in the patch. I'll try to go through sometime and read it with a tool that can ignore those. Sorry about that. Let me try and remove some. This stuff needs a refactor badly. I can't help myself. bq. I guess we still check if the handler is enabled, but it looks less clean than the other options we're exposing. Replication startup is complicated (I've yet to figure it out). Default in hbase is to start Replication. In the hollowed-out HRegionServer, we need Replication so we put up a Sink to receive Replication. On enable/disable of Replication, I'd like to do that in another issue. We have a million services/chores/managers and they all work different. Ideally, they'd all work the same and the flag to stop/start it would be same pattern no matter what the Service; but that is TODO I'd day. Idea behind this issue is to get the hbase-indexer folks something they can work with; my guess is they will come back with other stuff they'd like to be handled in subsequent issues before beta-1. Long-term, this masterless HRegionServer looks like it'll be useful testing HRegionServer in Isolation. bq. On the health check, can we get into a false positive state where a thread is null but shouldn't be? Is it better to check the configuration source of truth there instead? Perhaps. I'd like some more experience with hollow-RegionServer deploy first. Can fix in follow-on. bq. In the TestStandaloneRegionServer, let's put some of the cleanup (like the join) in either a finally block or a @After method so that we don't pollute other tests if we fail. Done (I tried to add more tests but short-circuiting RPC'ing messes up ability to test). > Accommodate the hbase-indexer/lily/SEP consumer deploy-type > --- > > Key: HBASE-18846 > URL: https://issues.apache.org/jira/browse/HBASE-18846 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18846.master.001.patch, > HBASE-18846.master.002.patch, HBASE-18846.master.003.patch, > HBASE-18846.master.004.patch, HBASE-18846.master.005.patch, > IndexerConnection.java, hbase-site.xml, javadoc.txt > > > This is a follow-on from HBASE-10504, Define a Replication Interface. There > we defined a new, flexible replication endpoint for others to implement but > it did little to help the case of the lily hbase-indexer. This issue takes up > the case of the hbase-indexer. > The hbase-indexer poses to hbase as a 'fake' peer cluster (For why > hbase-indexer is implemented so, the advantage to having the indexing done in > a separate process set that can be independently scaled, can participate in > the same security realm,
[jira] [Comment Edited] (HBASE-18846) Accommodate the hbase-indexer/lily/SEP consumer deploy-type
[ https://issues.apache.org/jira/browse/HBASE-18846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214095#comment-16214095 ] stack edited comment on HBASE-18846 at 10/22/17 4:23 AM: - bq. There's a lot of unrelated whitespace/formatting changes in the patch. I'll try to go through sometime and read it with a tool that can ignore those. Sorry about that. Let me try and remove some. This stuff needs a refactor badly. I can't help myself. bq. I guess we still check if the handler is enabled, but it looks less clean than the other options we're exposing. Replication startup is complicated (I've yet to figure it out). Default in hbase is to start Replication. In the hollowed-out HRegionServer, we need Replication so we put up a Sink to receive Replication. On enable/disable of Replication, I'd like to do that in another issue. We have a million services/chores/managers and they all work different. Ideally, they'd all work the same and the flag to stop/start it would be same pattern no matter what the Service; but that is TODO I'd day. Idea behind this issue is to get the hbase-indexer folks something they can work with; my guess is they will come back with other stuff they'd like to be handled in subsequent issues before beta-1. Long-term, this masterless HRegionServer looks like it'll be useful testing HRegionServer in Isolation. bq. On the health check, can we get into a false positive state where a thread is null but shouldn't be? Is it better to check the configuration source of truth there instead? Perhaps. I'd like some more experience with hollow-RegionServer deploy first. Can fix in follow-on. bq. In the TestStandaloneRegionServer, let's put some of the cleanup (like the join) in either a finally block or a @After method so that we don't pollute other tests if we fail. Done (I tried to add more tests but short-circuiting RPC'ing messes up ability to test). was (Author: stack): bq. There's a lot of unrelated whitespace/formatting changes in the patch. I'll try to go through sometime and read it with a tool that can ignore those. Sorry about that. Let me try and remove some. This stuff needs a refactor badly. I can't help myself. bq. I guess we still check if the handler is enabled, but it looks less clean than the other options we're exposing. Replication startup is complicated (I've yet to figure it out). Default in hbase is to start Replication. In the hollowed-out HRegionServer, we need Replication so we put up a Sink to receive Replication. On enable/disable of Replication, I'd like to do that in another issue. We have a million services/chores/managers and they all work different. Ideally, they'd all work the same and the flag to stop/start it would be same pattern no matter what the Service; but that is TODO I'd day. Idea behind this issue is to get the hbase-indexer folks something they can work with; my guess is they will come back with other stuff they'd like to be handled in subsequent issues before beta-1. Long-term, this masterless HRegionServer looks like it'll be useful testing HRegionServer in Isolation. bq. On the health check, can we get into a false positive state where a thread is null but shouldn't be? Is it better to check the configuration source of truth there instead? Perhaps. I'd like some more experience with hollow-RegionServer deploy first. Can fix in follow-on. bq. In the TestStandaloneRegionServer, let's put some of the cleanup (like the join) in either a finally block or a @After method so that we don't pollute other tests if we fail. Done. Added more tests too... for some of the other configs introduced here. > Accommodate the hbase-indexer/lily/SEP consumer deploy-type > --- > > Key: HBASE-18846 > URL: https://issues.apache.org/jira/browse/HBASE-18846 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18846.master.001.patch, > HBASE-18846.master.002.patch, HBASE-18846.master.003.patch, > HBASE-18846.master.004.patch, IndexerConnection.java, hbase-site.xml, > javadoc.txt > > > This is a follow-on from HBASE-10504, Define a Replication Interface. There > we defined a new, flexible replication endpoint for others to implement but > it did little to help the case of the lily hbase-indexer. This issue takes up > the case of the hbase-indexer. > The hbase-indexer poses to hbase as a 'fake' peer cluster (For why > hbase-indexer is implemented so, the advantage to having the indexing done in > a separate process set that can be independently scaled, can participate in > the same security realm, etc., see discussion in HBASE-10504). The > hbase-indexer will start up a cut-down "RegionServer" processes
[jira] [Comment Edited] (HBASE-18846) Accommodate the hbase-indexer/lily/SEP consumer deploy-type
[ https://issues.apache.org/jira/browse/HBASE-18846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214095#comment-16214095 ] stack edited comment on HBASE-18846 at 10/22/17 3:37 AM: - bq. There's a lot of unrelated whitespace/formatting changes in the patch. I'll try to go through sometime and read it with a tool that can ignore those. Sorry about that. Let me try and remove some. This stuff needs a refactor badly. I can't help myself. bq. I guess we still check if the handler is enabled, but it looks less clean than the other options we're exposing. Replication startup is complicated (I've yet to figure it out). Default in hbase is to start Replication. In the hollowed-out HRegionServer, we need Replication so we put up a Sink to receive Replication. On enable/disable of Replication, I'd like to do that in another issue. We have a million services/chores/managers and they all work different. Ideally, they'd all work the same and the flag to stop/start it would be same pattern no matter what the Service; but that is TODO I'd day. Idea behind this issue is to get the hbase-indexer folks something they can work with; my guess is they will come back with other stuff they'd like to be handled in subsequent issues before beta-1. Long-term, this masterless HRegionServer looks like it'll be useful testing HRegionServer in Isolation. bq. On the health check, can we get into a false positive state where a thread is null but shouldn't be? Is it better to check the configuration source of truth there instead? Perhaps. I'd like some more experience with hollow-RegionServer deploy first. Can fix in follow-on. bq. In the TestStandaloneRegionServer, let's put some of the cleanup (like the join) in either a finally block or a @After method so that we don't pollute other tests if we fail. Done. Added more tests too... for some of the other configs introduced here. was (Author: stack): bq. There's a lot of unrelated whitespace/formatting changes in the patch. I'll try to go through sometime and read it with a tool that can ignore those. Sorry about that. Let me try and remove some. This stuff needs a refactor. I can't help myself. bq. I guess we still check if the handler is enabled, but it looks less clean than the other options we're exposing. On Replication startup, I moved it under general startServices for now -- it'll be off in hollow-HRegionServer deploy ... Moved webui OUT of startServices because that already has its own enable/disable (Idea behind this issue is to get the hbase-indexer folks something they can work with; my guess is they will come back with other stuff they'd like to be handled in subsequent issues before beta-1. Long-term, this masterless HRegionServer looks like it'll be useful testing HRegionServer in Isolation. And whole world of start/stop services/managers/etc. is in bad need of cleanup. TODO). bq. On the health check, can we get into a false positive state where a thread is null but shouldn't be? Is it better to check the configuration source of truth there instead? Perhaps. I'd like some more experience with hollow-RegionServer deploy first. Can fix in follow-on. bq. In the TestStandaloneRegionServer, let's put some of the cleanup (like the join) in either a finally block or a @After method so that we don't pollute other tests if we fail. Done. Added more tests too... for some of the other configs introduced here. > Accommodate the hbase-indexer/lily/SEP consumer deploy-type > --- > > Key: HBASE-18846 > URL: https://issues.apache.org/jira/browse/HBASE-18846 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18846.master.001.patch, > HBASE-18846.master.002.patch, HBASE-18846.master.003.patch, > HBASE-18846.master.004.patch, IndexerConnection.java, hbase-site.xml, > javadoc.txt > > > This is a follow-on from HBASE-10504, Define a Replication Interface. There > we defined a new, flexible replication endpoint for others to implement but > it did little to help the case of the lily hbase-indexer. This issue takes up > the case of the hbase-indexer. > The hbase-indexer poses to hbase as a 'fake' peer cluster (For why > hbase-indexer is implemented so, the advantage to having the indexing done in > a separate process set that can be independently scaled, can participate in > the same security realm, etc., see discussion in HBASE-10504). The > hbase-indexer will start up a cut-down "RegionServer" processes that are just > an instance of hbase RpcServer hosting an AdminProtos Service. They make > themselves 'appear' to the Replication Source by hoisting up an ephemeral > znode 'registering' as a RegionServer. The source
[jira] [Comment Edited] (HBASE-18846) Accommodate the hbase-indexer/lily/SEP consumer deploy-type
[
https://issues.apache.org/jira/browse/HBASE-18846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214095#comment-16214095
]
stack edited comment on HBASE-18846 at 10/21/17 8:50 PM:
-
bq. There's a lot of unrelated whitespace/formatting changes in the patch. I'll
try to go through sometime and read it with a tool that can ignore those.
Sorry about that. Let me try and remove some. This stuff needs a refactor. I
can't help myself.
bq. I guess we still check if the handler is enabled, but it looks less clean
than the other options we're exposing.
On Replication startup, I moved it under general startServices for now -- it'll
be off in hollow-HRegionServer deploy ... Moved webui OUT of startServices
because that already has its own enable/disable (Idea behind this issue is to
get the hbase-indexer folks something they can work with; my guess is they will
come back with other stuff they'd like to be handled in subsequent issues
before beta-1. Long-term, this masterless HRegionServer looks like it'll be
useful testing HRegionServer in Isolation. And whole world of start/stop
services/managers/etc. is in bad need of cleanup. TODO).
bq. On the health check, can we get into a false positive state where a thread
is null but shouldn't be? Is it better to check the configuration source of
truth there instead?
Perhaps. I'd like some more experience with hollow-RegionServer deploy first.
Can fix in follow-on.
bq. In the TestStandaloneRegionServer, let's put some of the cleanup (like the
join) in either a finally block or a @After method so that we don't pollute
other tests if we fail.
Done. Added more tests too... for some of the other configs introduced here.
was (Author: stack):
bq. There's a lot of unrelated whitespace/formatting changes in the patch. I'll
try to go through sometime and read it with a tool that can ignore those.
Sorry about that. Let me try and remove some. This stuff needs a refactor. I
can't help myself.
bq. I guess we still check if the handler is enabled, but it looks less clean
than the other options we're exposing.
On Replication startup, see note at tail of setupWALAndReplication. There is
some circular dependency between Replication and WAL. I'd suggest untangle is
outside scope of this issue (Idea behind this issue is to get the hbase-indexer
folks something they can work with; my guess is they will come back with other
stuff they'd like to be handled in subsequent issues before beta-1. Long-term,
this masterless HRegionServer looks like it'll be useful testing HRegionServer
in Isolation. TODO).
bq. On the health check, can we get into a false positive state where a thread
is null but shouldn't be? Is it better to check the configuration source of
truth there instead?
Perhaps. I'd like some more experience with hollow-RegionServer deploy first.
Can fix in follow-on.
bq. In the TestStandaloneRegionServer, let's put some of the cleanup (like the
join) in either a finally block or a @After method so that we don't pollute
other tests if we fail.
Done. Added more tests too... for some of the other configs introduced here.
> Accommodate the hbase-indexer/lily/SEP consumer deploy-type
> ---
>
> Key: HBASE-18846
> URL: https://issues.apache.org/jira/browse/HBASE-18846
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0-alpha-4
>
> Attachments: HBASE-18846.master.001.patch,
> HBASE-18846.master.002.patch, HBASE-18846.master.003.patch,
> HBASE-18846.master.004.patch, IndexerConnection.java, hbase-site.xml,
> javadoc.txt
>
>
> This is a follow-on from HBASE-10504, Define a Replication Interface. There
> we defined a new, flexible replication endpoint for others to implement but
> it did little to help the case of the lily hbase-indexer. This issue takes up
> the case of the hbase-indexer.
> The hbase-indexer poses to hbase as a 'fake' peer cluster (For why
> hbase-indexer is implemented so, the advantage to having the indexing done in
> a separate process set that can be independently scaled, can participate in
> the same security realm, etc., see discussion in HBASE-10504). The
> hbase-indexer will start up a cut-down "RegionServer" processes that are just
> an instance of hbase RpcServer hosting an AdminProtos Service. They make
> themselves 'appear' to the Replication Source by hoisting up an ephemeral
> znode 'registering' as a RegionServer. The source cluster then streams
> WALEdits to the Admin Protos method:
> {code}
> public ReplicateWALEntryResponse replicateWALEntry(final RpcController
> controller,
> final ReplicateWALEntryRequest request) throws ServiceException {
> {code}
> The hbase-indexer relies on other hbase internals like Server so
[jira] [Comment Edited] (HBASE-18846) Accommodate the hbase-indexer/lily/SEP consumer deploy-type
[
https://issues.apache.org/jira/browse/HBASE-18846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213760#comment-16213760
]
stack edited comment on HBASE-18846 at 10/21/17 7:45 PM:
-
No. Did not mean to include the hbase-site and sample indexer. Thanks.
Fixed the class comment (s/emasculated/hollow/).
Will put up new patch to address your other comments. Thanks for taking a look
[~mdrob]
was (Author: stack):
No. Did not mean to include the hbase-site and sample indexer. Thanks.
Fixed the release note (s/emasculated/hollow/).
Will put up new patch to address your other comments. Thanks for taking a look
[~mdrob]
> Accommodate the hbase-indexer/lily/SEP consumer deploy-type
> ---
>
> Key: HBASE-18846
> URL: https://issues.apache.org/jira/browse/HBASE-18846
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0-alpha-4
>
> Attachments: HBASE-18846.master.001.patch,
> HBASE-18846.master.002.patch, HBASE-18846.master.003.patch,
> HBASE-18846.master.004.patch, IndexerConnection.java, hbase-site.xml,
> javadoc.txt
>
>
> This is a follow-on from HBASE-10504, Define a Replication Interface. There
> we defined a new, flexible replication endpoint for others to implement but
> it did little to help the case of the lily hbase-indexer. This issue takes up
> the case of the hbase-indexer.
> The hbase-indexer poses to hbase as a 'fake' peer cluster (For why
> hbase-indexer is implemented so, the advantage to having the indexing done in
> a separate process set that can be independently scaled, can participate in
> the same security realm, etc., see discussion in HBASE-10504). The
> hbase-indexer will start up a cut-down "RegionServer" processes that are just
> an instance of hbase RpcServer hosting an AdminProtos Service. They make
> themselves 'appear' to the Replication Source by hoisting up an ephemeral
> znode 'registering' as a RegionServer. The source cluster then streams
> WALEdits to the Admin Protos method:
> {code}
> public ReplicateWALEntryResponse replicateWALEntry(final RpcController
> controller,
> final ReplicateWALEntryRequest request) throws ServiceException {
> {code}
> The hbase-indexer relies on other hbase internals like Server so it can get a
> ZooKeeperWatcher instance and know the 'name' to use for this cut-down server.
> Thoughts on how to proceed include:
>
> * Better formalize its current digestion of hbase internals; make it so
> rpcserver is allowed to be used by others, etc. This would be hard to do
> given they use basics like Server, Protobuf serdes for WAL types, and
> AdminProtos Service. Any change in this wide API breaks (again)
> hbase-indexer. We have made a 'channel' for Coprocessor Endpoints so they
> continue to work though they use 'internal' types. They can use protos in
> hbase-protocol. hbase-protocol protos are in a limbo currently where they are
> sort-of 'public'; a TODO. Perhaps the hbase-indexer could do similar relying
> on the hbase-protocol (pb2.5) content and we could do something to reveal
> rpcserver and zk for hbase-indexer safe use.
> * Start an actual RegionServer only have it register the AdminProtos Service
> only -- not ClientProtos and the Service that does Master interaction, etc.
> [I checked, this is not as easy to do as I at first thought -- St.Ack] Then
> have the hbase-indexer implement an AdminCoprocessor to override the
> replicateWALEntry method (the Admin CP implementation may need work). This
> would narrow the hbase-indexer exposure to that of the Admin Coprocessor
> Interface
> * Over in HBASE-10504, [~enis] suggested "... if we want to provide
> isolation for the replication services in hbase, we can have a simple host as
> another daemon which hosts the ReplicationEndpoint implementation. RS's will
> use a built-in RE to send the edits to this layer, and the host will delegate
> it to the RE implementation. The flow would be something like: RS --> RE
> inside RS --> Host daemon for RE --> Actual RE implementation --> third party
> system..."
>
> Other crazy notions occur including the setup of an Admin Interface
> Coprocessor Endpoint. A new ReplicationEndpoint would feed the replication
> stream to the remote cluster via the CPEP registered channel.
> But time is short. Hopefully we can figure something that will work in 2.0
> timeframe w/o too much code movement.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
