[jira] [Commented] (KNOX-1041) High Availability Support For Apache SOLR, HBase & Kafka
[ https://issues.apache.org/jira/browse/KNOX-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16219337#comment-16219337 ] ASF subversion and git services commented on KNOX-1041: --- Commit a08aaf742a97a3c35c94e28406fc4b6ef3184005 in knox's branch refs/heads/KNOX-998-Package_Restructuring from [~moresandeep] [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=a08aaf7 ] KNOX-1041 - High Availability Support For Apache SOLR, HBase & Kafka (Rick Kellogg via Sandeep More) > High Availability Support For Apache SOLR, HBase & Kafka > > > Key: KNOX-1041 > URL: https://issues.apache.org/jira/browse/KNOX-1041 > Project: Apache Knox > Issue Type: New Feature > Components: Server >Affects Versions: 0.12.0, 0.13.0 >Reporter: Rick Kellogg >Assignee: Rick Kellogg > Fix For: 0.14.0 > > Attachments: KNOX-1041.patch > > > Provide high-availability/fail-over between Knox and SOLR/HBase/Kafka using > the existing DefaultHaDispatch mechanism and a customized URLManager > implementation with knowledge of active hosts in Zookeeper. > When SOLR Cloud is used the active hosts are stored in Zookeeper under the > /live_nodes path. The attached custom URLManager implementation queries > Zookeeper for the active hosts upon startup. In the event of fail-over, it > updates the internal list of hosts. > The HS2ZookeeperURLManager implementation used to provide similar > functionality for Hive was used as a starting point. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KNOX-1041) High Availability Support For Apache SOLR, HBase & Kafka
[ https://issues.apache.org/jira/browse/KNOX-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212721#comment-16212721 ] ASF subversion and git services commented on KNOX-1041: --- Commit a08aaf742a97a3c35c94e28406fc4b6ef3184005 in knox's branch refs/heads/master from [~moresandeep] [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=a08aaf7 ] KNOX-1041 - High Availability Support For Apache SOLR, HBase & Kafka (Rick Kellogg via Sandeep More) > High Availability Support For Apache SOLR, HBase & Kafka > > > Key: KNOX-1041 > URL: https://issues.apache.org/jira/browse/KNOX-1041 > Project: Apache Knox > Issue Type: New Feature > Components: Server >Affects Versions: 0.12.0, 0.13.0 >Reporter: Rick Kellogg >Assignee: Rick Kellogg > Fix For: 0.14.0 > > Attachments: KNOX-1041.patch > > > Provide high-availability/fail-over between Knox and SOLR/HBase/Kafka using > the existing DefaultHaDispatch mechanism and a customized URLManager > implementation with knowledge of active hosts in Zookeeper. > When SOLR Cloud is used the active hosts are stored in Zookeeper under the > /live_nodes path. The attached custom URLManager implementation queries > Zookeeper for the active hosts upon startup. In the event of fail-over, it > updates the internal list of hosts. > The HS2ZookeeperURLManager implementation used to provide similar > functionality for Hive was used as a starting point. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KNOX-1041) High Availability Support For Apache SOLR, HBase & Kafka
[ https://issues.apache.org/jira/browse/KNOX-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212723#comment-16212723 ] Sandeep More commented on KNOX-1041: Thanks the contribution and being patient [~rkellogg] :) I have committed the patch to master. > High Availability Support For Apache SOLR, HBase & Kafka > > > Key: KNOX-1041 > URL: https://issues.apache.org/jira/browse/KNOX-1041 > Project: Apache Knox > Issue Type: New Feature > Components: Server >Affects Versions: 0.12.0, 0.13.0 >Reporter: Rick Kellogg >Assignee: Rick Kellogg > Fix For: 0.14.0 > > Attachments: KNOX-1041.patch > > > Provide high-availability/fail-over between Knox and SOLR/HBase/Kafka using > the existing DefaultHaDispatch mechanism and a customized URLManager > implementation with knowledge of active hosts in Zookeeper. > When SOLR Cloud is used the active hosts are stored in Zookeeper under the > /live_nodes path. The attached custom URLManager implementation queries > Zookeeper for the active hosts upon startup. In the event of fail-over, it > updates the internal list of hosts. > The HS2ZookeeperURLManager implementation used to provide similar > functionality for Hive was used as a starting point. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KNOX-1041) High Availability Support For Apache SOLR, HBase & Kafka
[ https://issues.apache.org/jira/browse/KNOX-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16186307#comment-16186307 ] Rick Kellogg commented on KNOX-1041: Could you try it without my unit tests included? I see an error in the HS2ZookeeperURLManagerTest which was not modified. It might be something with your environment. Based on the error message it seems your hostname is not configured properly. Check the /etc/hostname file. Can you provide retails on your testing environments? OS, JDK, etc? Will try to replicate this weekend. > High Availability Support For Apache SOLR, HBase & Kafka > > > Key: KNOX-1041 > URL: https://issues.apache.org/jira/browse/KNOX-1041 > Project: Apache Knox > Issue Type: New Feature > Components: Server >Affects Versions: 0.12.0, 0.13.0 >Reporter: Rick Kellogg >Assignee: Rick Kellogg > Fix For: 0.14.0 > > Attachments: KNOX-1041.patch > > > Provide high-availability/fail-over between Knox and SOLR/HBase/Kafka using > the existing DefaultHaDispatch mechanism and a customized URLManager > implementation with knowledge of active hosts in Zookeeper. > When SOLR Cloud is used the active hosts are stored in Zookeeper under the > /live_nodes path. The attached custom URLManager implementation queries > Zookeeper for the active hosts upon startup. In the event of fail-over, it > updates the internal list of hosts. > The HS2ZookeeperURLManager implementation used to provide similar > functionality for Hive was used as a starting point. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KNOX-1041) High Availability Support For Apache SOLR, HBase & Kafka
[ https://issues.apache.org/jira/browse/KNOX-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174747#comment-16174747 ] Phil Zampino commented on KNOX-1041: The changes to BaseZookeeperURLManager look good. Thanks. > High Availability Support For Apache SOLR, HBase & Kafka > > > Key: KNOX-1041 > URL: https://issues.apache.org/jira/browse/KNOX-1041 > Project: Apache Knox > Issue Type: New Feature > Components: Server >Affects Versions: 0.12.0, 0.13.0 >Reporter: Rick Kellogg >Assignee: Rick Kellogg > Fix For: 0.14.0 > > Attachments: KNOX-1041.patch > > > Provide high-availability/fail-over between Knox and SOLR/HBase/Kafka using > the existing DefaultHaDispatch mechanism and a customized URLManager > implementation with knowledge of active hosts in Zookeeper. > When SOLR Cloud is used the active hosts are stored in Zookeeper under the > /live_nodes path. The attached custom URLManager implementation queries > Zookeeper for the active hosts upon startup. In the event of fail-over, it > updates the internal list of hosts. > The HS2ZookeeperURLManager implementation used to provide similar > functionality for Hive was used as a starting point. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KNOX-1041) High Availability Support For Apache SOLR, HBase & Kafka
[ https://issues.apache.org/jira/browse/KNOX-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174149#comment-16174149 ] Rick Kellogg commented on KNOX-1041: Scroll to the bottom of the patch. They are there. If not I can regenerate tomorrow. > High Availability Support For Apache SOLR, HBase & Kafka > > > Key: KNOX-1041 > URL: https://issues.apache.org/jira/browse/KNOX-1041 > Project: Apache Knox > Issue Type: New Feature > Components: Server >Affects Versions: 0.12.0, 0.13.0 >Reporter: Rick Kellogg >Assignee: Rick Kellogg > Fix For: 0.14.0 > > Attachments: KNOX-1041.patch > > > Provide high-availability/fail-over between Knox and SOLR/HBase/Kafka using > the existing DefaultHaDispatch mechanism and a customized URLManager > implementation with knowledge of active hosts in Zookeeper. > When SOLR Cloud is used the active hosts are stored in Zookeeper under the > /live_nodes path. The attached custom URLManager implementation queries > Zookeeper for the active hosts upon startup. In the event of fail-over, it > updates the internal list of hosts. > The HS2ZookeeperURLManager implementation used to provide similar > functionality for Hive was used as a starting point. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KNOX-1041) High Availability Support For Apache SOLR, HBase & Kafka
[ https://issues.apache.org/jira/browse/KNOX-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174138#comment-16174138 ] Phil Zampino commented on KNOX-1041: I'm not seeing the changes in the attached patch. > High Availability Support For Apache SOLR, HBase & Kafka > > > Key: KNOX-1041 > URL: https://issues.apache.org/jira/browse/KNOX-1041 > Project: Apache Knox > Issue Type: New Feature > Components: Server >Affects Versions: 0.12.0, 0.13.0 >Reporter: Rick Kellogg >Assignee: Rick Kellogg > Fix For: 0.14.0 > > Attachments: KNOX-1041.patch > > > Provide high-availability/fail-over between Knox and SOLR/HBase/Kafka using > the existing DefaultHaDispatch mechanism and a customized URLManager > implementation with knowledge of active hosts in Zookeeper. > When SOLR Cloud is used the active hosts are stored in Zookeeper under the > /live_nodes path. The attached custom URLManager implementation queries > Zookeeper for the active hosts upon startup. In the event of fail-over, it > updates the internal list of hosts. > The HS2ZookeeperURLManager implementation used to provide similar > functionality for Hive was used as a starting point. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KNOX-1041) High Availability Support For Apache SOLR, HBase & Kafka
[ https://issues.apache.org/jira/browse/KNOX-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173813#comment-16173813 ] Phil Zampino commented on KNOX-1041: There is still some concern around thread-safety. In fact, I've filed KNOX-1054 to address the similar issue in DefaultURLManager. Basically, even though the underlying data structure is a ConcurrentLinkedQueue, the use of that queue is not thread-safe. At a minimum, the "read" methods must be synchronized, such that they're subject to the same lock as the "write" methods. > High Availability Support For Apache SOLR, HBase & Kafka > > > Key: KNOX-1041 > URL: https://issues.apache.org/jira/browse/KNOX-1041 > Project: Apache Knox > Issue Type: New Feature > Components: Server >Affects Versions: 0.12.0, 0.13.0 >Reporter: Rick Kellogg >Assignee: Rick Kellogg > Fix For: 0.14.0 > > Attachments: KNOX-1041.patch > > > Provide high-availability/fail-over between Knox and SOLR/HBase/Kafka using > the existing DefaultHaDispatch mechanism and a customized URLManager > implementation with knowledge of active hosts in Zookeeper. > When SOLR Cloud is used the active hosts are stored in Zookeeper under the > /live_nodes path. The attached custom URLManager implementation queries > Zookeeper for the active hosts upon startup. In the event of fail-over, it > updates the internal list of hosts. > The HS2ZookeeperURLManager implementation used to provide similar > functionality for Hive was used as a starting point. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KNOX-1041) High Availability Support For Apache SOLR, HBase & Kafka
[
https://issues.apache.org/jira/browse/KNOX-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173656#comment-16173656
]
Phil Zampino commented on KNOX-1041:
In BaseZookeeperURLManager#getActiveURL():
_+ String result = null;
+
+ if (!urls.isEmpty()) {
+ result = this.urls.peek();
+ }
+_
Wouldn't the following have the same effect?
_String result = this.urls.peek();_
Based on the javadoc for ConcurrentLinkedQueue:
_public E peek()
Description copied from interface: Queue
Retrieves, but does not remove, the head of this queue, or *returns null if
this queue is empty*._
> High Availability Support For Apache SOLR, HBase & Kafka
>
>
> Key: KNOX-1041
> URL: https://issues.apache.org/jira/browse/KNOX-1041
> Project: Apache Knox
> Issue Type: New Feature
> Components: Server
>Affects Versions: 0.12.0, 0.13.0
>Reporter: Rick Kellogg
>Assignee: Rick Kellogg
> Fix For: 0.14.0
>
> Attachments: KNOX-1041.patch
>
>
> Provide high-availability/fail-over between Knox and SOLR/HBase/Kafka using
> the existing DefaultHaDispatch mechanism and a customized URLManager
> implementation with knowledge of active hosts in Zookeeper.
> When SOLR Cloud is used the active hosts are stored in Zookeeper under the
> /live_nodes path. The attached custom URLManager implementation queries
> Zookeeper for the active hosts upon startup. In the event of fail-over, it
> updates the internal list of hosts.
> The HS2ZookeeperURLManager implementation used to provide similar
> functionality for Hive was used as a starting point.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KNOX-1041) High Availability Support For Apache SOLR, HBase & Kafka
[ https://issues.apache.org/jira/browse/KNOX-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173528#comment-16173528 ] Phil Zampino commented on KNOX-1041: [~rkellogg] I don't see the attachment at all now. > High Availability Support For Apache SOLR, HBase & Kafka > > > Key: KNOX-1041 > URL: https://issues.apache.org/jira/browse/KNOX-1041 > Project: Apache Knox > Issue Type: New Feature > Components: Server >Affects Versions: 0.12.0, 0.13.0 >Reporter: Rick Kellogg >Assignee: Rick Kellogg > Fix For: 0.14.0 > > > Provide high-availability/fail-over between Knox and SOLR/HBase/Kafka using > the existing DefaultHaDispatch mechanism and a customized URLManager > implementation with knowledge of active hosts in Zookeeper. > When SOLR Cloud is used the active hosts are stored in Zookeeper under the > /live_nodes path. The attached custom URLManager implementation queries > Zookeeper for the active hosts upon startup. In the event of fail-over, it > updates the internal list of hosts. > The HS2ZookeeperURLManager implementation used to provide similar > functionality for Hive was used as a starting point. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KNOX-1041) High Availability Support For Apache SOLR, HBase & Kafka
[ https://issues.apache.org/jira/browse/KNOX-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173517#comment-16173517 ] Rick Kellogg commented on KNOX-1041: Agree with all your comments. Updated patch provided. > High Availability Support For Apache SOLR, HBase & Kafka > > > Key: KNOX-1041 > URL: https://issues.apache.org/jira/browse/KNOX-1041 > Project: Apache Knox > Issue Type: New Feature > Components: Server >Affects Versions: 0.12.0, 0.13.0 >Reporter: Rick Kellogg >Assignee: Rick Kellogg > Fix For: 0.14.0 > > Attachments: KNOX-1041.patch > > > Provide high-availability/fail-over between Knox and SOLR/HBase/Kafka using > the existing DefaultHaDispatch mechanism and a customized URLManager > implementation with knowledge of active hosts in Zookeeper. > When SOLR Cloud is used the active hosts are stored in Zookeeper under the > /live_nodes path. The attached custom URLManager implementation queries > Zookeeper for the active hosts upon startup. In the event of fail-over, it > updates the internal list of hosts. > The HS2ZookeeperURLManager implementation used to provide similar > functionality for Hive was used as a starting point. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KNOX-1041) High Availability Support For Apache SOLR, HBase & Kafka
[
https://issues.apache.org/jira/browse/KNOX-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173366#comment-16173366
]
Phil Zampino commented on KNOX-1041:
In BaseZookeeperURLManager.java, I think the class comment has a typo ("common
delimited list"); I'm assuming you meant "comma-delimited list".
constructURL(String) comment incldues "http://host:post/solr;; Should that be
"port" instead of "post"?
Also, could the synchronization be more granular? Many methods are
synchronized, which locks the whole object. Some methods, like
setActiveURL(String) don't need any synchronization (unless there is an
intention to replace the UnsupportedOperationException in the future).
> High Availability Support For Apache SOLR, HBase & Kafka
>
>
> Key: KNOX-1041
> URL: https://issues.apache.org/jira/browse/KNOX-1041
> Project: Apache Knox
> Issue Type: New Feature
> Components: Server
>Affects Versions: 0.12.0, 0.13.0
>Reporter: Rick Kellogg
>Assignee: Rick Kellogg
> Fix For: 0.14.0
>
> Attachments: KNOX-1041.patch
>
>
> Provide high-availability/fail-over between Knox and SOLR/HBase/Kafka using
> the existing DefaultHaDispatch mechanism and a customized URLManager
> implementation with knowledge of active hosts in Zookeeper.
> When SOLR Cloud is used the active hosts are stored in Zookeeper under the
> /live_nodes path. The attached custom URLManager implementation queries
> Zookeeper for the active hosts upon startup. In the event of fail-over, it
> updates the internal list of hosts.
> The HS2ZookeeperURLManager implementation used to provide similar
> functionality for Hive was used as a starting point.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KNOX-1041) High Availability Support For Apache SOLR, HBase & Kafka
[ https://issues.apache.org/jira/browse/KNOX-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173283#comment-16173283 ] Rick Kellogg commented on KNOX-1041: For both HBase and Kafka, the REST Server is not registered within Zookeeper. Instead we look up the location of the HBase RegionServer (/hbase-unsecure) or Kafka (/broker/ids). Using the host information found a light weight ping style operation is done to determine if the REST Server is operational. In the future, it would be preferable to read live host information directly from Zookeeper. Enhancement requests have been made to both projects. See links above. > High Availability Support For Apache SOLR, HBase & Kafka > > > Key: KNOX-1041 > URL: https://issues.apache.org/jira/browse/KNOX-1041 > Project: Apache Knox > Issue Type: New Feature > Components: Server >Affects Versions: 0.12.0, 0.13.0 >Reporter: Rick Kellogg >Assignee: Rick Kellogg > Fix For: 0.14.0 > > Attachments: KNOX-1041.patch > > > Provide high-availability/fail-over between Knox and SOLR/HBase/Kafka using > the existing DefaultHaDispatch mechanism and a customized URLManager > implementation with knowledge of active hosts in Zookeeper. > When SOLR Cloud is used the active hosts are stored in Zookeeper under the > /live_nodes path. The attached custom URLManager implementation queries > Zookeeper for the active hosts upon startup. In the event of fail-over, it > updates the internal list of hosts. > The HS2ZookeeperURLManager implementation used to provide similar > functionality for Hive was used as a starting point. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KNOX-1041) High Availability Support For Apache SOLR, HBase & Kafka
[
https://issues.apache.org/jira/browse/KNOX-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173270#comment-16173270
]
Rick Kellogg commented on KNOX-1041:
Example configuration for KAFKA:
{noformat}
ha
HaProvider
true
KAFKA
maxFailoverAttempts=3;failoverSleep=1000;enabled=true;zookeeperEnsemble=machine1:2181,machine2:2181,machine3:2181
{noformat}
Example configuration for HBase:
{noformat}
ha
HaProvider
true
WEBHBASE
maxFailoverAttempts=3;failoverSleep=1000;enabled=true;zookeeperEnsemble=machine1:2181,machine2:2181,machine3:2181
{noformat}
> High Availability Support For Apache SOLR, HBase & Kafka
>
>
> Key: KNOX-1041
> URL: https://issues.apache.org/jira/browse/KNOX-1041
> Project: Apache Knox
> Issue Type: New Feature
> Components: Server
>Affects Versions: 0.12.0, 0.13.0
>Reporter: Rick Kellogg
>Assignee: Rick Kellogg
> Fix For: 0.14.0
>
> Attachments: KNOX-1041.patch
>
>
> Provide high-availability/fail-over between Knox and SOLR/HBase/Kafka using
> the existing DefaultHaDispatch mechanism and a customized URLManager
> implementation with knowledge of active hosts in Zookeeper.
> When SOLR Cloud is used the active hosts are stored in Zookeeper under the
> /live_nodes path. The attached custom URLManager implementation queries
> Zookeeper for the active hosts upon startup. In the event of fail-over, it
> updates the internal list of hosts.
> The HS2ZookeeperURLManager implementation used to provide similar
> functionality for Hive was used as a starting point.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
