[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[ https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731386#comment-15731386 ] Kunal Khatua commented on DRILL-5015: - [~khfaraaz] Please verify and close. > As per documentation, when issuing a list of drillbits in the connection > string, we always attempt to connect only to the first one > --- > > Key: DRILL-5015 > URL: https://issues.apache.org/jira/browse/DRILL-5015 > Project: Apache Drill > Issue Type: Bug > Components: Client - JDBC >Affects Versions: 1.8.0, 1.9.0 >Reporter: Sorabh Hamirwasia >Assignee: Sudheesh Katkam > Labels: ready-to-commit > Fix For: 1.10.0 > > > When trying to connect to a Drill cluster by specifying more than 1 drillbits > to connect to, we always attempt to connect to only the first drillbit. > As an example, we tested against a pair of drillbits, but we always connect > to the first entry in the CSV list by querying for the 'current' drillbit. > The remaining entries are never attempted. > [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u > "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f > whereAmI.q | grep -v logback > 1/1 select * from sys.drillbits where `current`; > +-++---++--+ > |hostname | user_port | control_port | data_port | current | > +-++---++--+ > | pssc-61.qa.lab | 31010 | 31011 | 31012 | true | > +-++---++--+ > 1 row selected (0.265 seconds) > Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl > apache drill 1.8.0 > "a little sql for your nosql" > This property is meant for use by clients when not wanting to overload the ZK > for fetching a list of existing Drillbits, but the behaviour doesn't match > the documentation. > [Making a Direct Drillbit Connection | > https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection > ] > We need to randomly shuffle between this list and If an entry in the shuffled > list is unreachable, we need to try for the next entry in the list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[ https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15723033#comment-15723033 ] ASF GitHub Bot commented on DRILL-5015: --- Github user asfgit closed the pull request at: https://github.com/apache/drill/pull/648 > As per documentation, when issuing a list of drillbits in the connection > string, we always attempt to connect only to the first one > --- > > Key: DRILL-5015 > URL: https://issues.apache.org/jira/browse/DRILL-5015 > Project: Apache Drill > Issue Type: Bug > Components: Client - JDBC >Affects Versions: 1.8.0, 1.9.0 >Reporter: Sorabh Hamirwasia >Assignee: Sudheesh Katkam > Labels: ready-to-commit > Fix For: 1.10.0 > > > When trying to connect to a Drill cluster by specifying more than 1 drillbits > to connect to, we always attempt to connect to only the first drillbit. > As an example, we tested against a pair of drillbits, but we always connect > to the first entry in the CSV list by querying for the 'current' drillbit. > The remaining entries are never attempted. > [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u > "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f > whereAmI.q | grep -v logback > 1/1 select * from sys.drillbits where `current`; > +-++---++--+ > |hostname | user_port | control_port | data_port | current | > +-++---++--+ > | pssc-61.qa.lab | 31010 | 31011 | 31012 | true | > +-++---++--+ > 1 row selected (0.265 seconds) > Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl > apache drill 1.8.0 > "a little sql for your nosql" > This property is meant for use by clients when not wanting to overload the ZK > for fetching a list of existing Drillbits, but the behaviour doesn't match > the documentation. > [Making a Direct Drillbit Connection | > https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection > ] > We need to randomly shuffle between this list and If an entry in the shuffled > list is unreachable, we need to try for the next entry in the list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[ https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15713485#comment-15713485 ] ASF GitHub Bot commented on DRILL-5015: --- Github user sudheeshkatkam commented on the issue: https://github.com/apache/drill/pull/648 +1 > As per documentation, when issuing a list of drillbits in the connection > string, we always attempt to connect only to the first one > --- > > Key: DRILL-5015 > URL: https://issues.apache.org/jira/browse/DRILL-5015 > Project: Apache Drill > Issue Type: Bug > Components: Client - JDBC >Affects Versions: 1.8.0, 1.9.0 >Reporter: Sorabh Hamirwasia >Assignee: Sudheesh Katkam > Labels: ready-to-commit > > When trying to connect to a Drill cluster by specifying more than 1 drillbits > to connect to, we always attempt to connect to only the first drillbit. > As an example, we tested against a pair of drillbits, but we always connect > to the first entry in the CSV list by querying for the 'current' drillbit. > The remaining entries are never attempted. > [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u > "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f > whereAmI.q | grep -v logback > 1/1 select * from sys.drillbits where `current`; > +-++---++--+ > |hostname | user_port | control_port | data_port | current | > +-++---++--+ > | pssc-61.qa.lab | 31010 | 31011 | 31012 | true | > +-++---++--+ > 1 row selected (0.265 seconds) > Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl > apache drill 1.8.0 > "a little sql for your nosql" > This property is meant for use by clients when not wanting to overload the ZK > for fetching a list of existing Drillbits, but the behaviour doesn't match > the documentation. > [Making a Direct Drillbit Connection | > https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection > ] > We need to randomly shuffle between this list and If an entry in the shuffled > list is unreachable, we need to try for the next entry in the list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15706598#comment-15706598
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r90089781
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +224,100 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Populates the endpointlist with drillbits information provided in the
connection string by client.
+ * For direct connection we can have connection string with drillbit
property as below:
+ *
+ * drillbit=ip
+ * use the ip specified as the Foreman ip with default port in
config file
+ * drillbit=ip:port
+ * use the ip and port specified as the Foreman ip and port
+ * drillbit=ip1:port1,ip2:port2,...
+ * randomly select the ip and port pair from the specified list as
the Foreman ip and port.
+ *
+ *
+ * @param drillbits string with drillbit value provided in connection
string
+ * @param defaultUserPort string with default userport of drillbit
specified in config file
+ * @return list of drillbit endpoints parsed from connection string
+ * @throws InvalidConnectionInfoException if the connection string has
invalid or no drillbit information
+ */
+ static List parseAndVerifyEndpoints(String drillbits,
String defaultUserPort)
+throws InvalidConnectionInfoException {
+// If no drillbits is provided then throw exception
+drillbits = drillbits.trim();
+if (drillbits.isEmpty()) {
+ throw new InvalidConnectionInfoException("No drillbit information
specified in the connection string");
+}
+
+ArrayList endpointList = new ArrayList<>();
+final String[] connectInfo = drillbits.split(",");
+
+// Fetch ip address and port information for each drillbit and
populate the list
+for (String drillbit : connectInfo) {
+
+ // Trim all the empty spaces and check if the entry is empty string.
+ // Ignore the empty ones.
+ drillbit = drillbit.trim();
+
+ if (!drillbit.isEmpty()) {
+// Verify if we have only ":" or only ":port" pattern
+if (drillbit.charAt(0) == ':') {
+ // Invalid drillbit information
+ throw new InvalidConnectionInfoException("Malformed connection
string with drillbit hostname or " +
+ "hostaddress missing
for an entry: " + drillbit);
+}
+
+// We are now sure that each ip:port entry will have both the
values atleast once.
+// Split each drillbit connection string to get ip address and
port value
+final String[] drillbitInfo = drillbit.split(":");
+
+// Check if we have more than one port
+if (drillbitInfo.length > 2) {
+ throw new InvalidConnectionInfoException("Malformed connection
string with more than one port in a " +
+ "drillbit entry: " +
drillbit);
+}
+
+// At this point we are sure that drillbitInfo has atleast
hostname or host address
+// trim all the empty spaces which might be present in front of
hostname or
+// host address information
+final String ipAddress = drillbitInfo[0].trim();
+String port = defaultUserPort;
+
+if (drillbitInfo.length == 2) {
+ // We have a port value also given by user. trim all the empty
spaces between : and port value before
+ // validating the correctness of value.
+ port = drillbitInfo[1].trim();
+}
+
+try {
+ final DrillbitEndpoint endpoint = DrillbitEndpoint.newBuilder()
+.setAddress(ipAddress)
+
.setUserPort(Integer.parseInt(port))
+.build();
+
+ endpointList.add(endpoint);
+} catch (NumberFormatException e) {
+ throw new InvalidConnectionInfoException("Malformed port value
in entry: " + ipAddress + ":" + port + " " +
+ "passed in connection
string");
+}
+ }
+}
+if(endpointList.size() == 0){
--- End diff --
fixed
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> -
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15706603#comment-15706603
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r90089750
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +224,100 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Populates the endpointlist with drillbits information provided in the
connection string by client.
+ * For direct connection we can have connection string with drillbit
property as below:
+ *
+ * drillbit=ip
+ * use the ip specified as the Foreman ip with default port in
config file
+ * drillbit=ip:port
+ * use the ip and port specified as the Foreman ip and port
+ * drillbit=ip1:port1,ip2:port2,...
+ * randomly select the ip and port pair from the specified list as
the Foreman ip and port.
+ *
+ *
+ * @param drillbits string with drillbit value provided in connection
string
+ * @param defaultUserPort string with default userport of drillbit
specified in config file
+ * @return list of drillbit endpoints parsed from connection string
+ * @throws InvalidConnectionInfoException if the connection string has
invalid or no drillbit information
+ */
+ static List parseAndVerifyEndpoints(String drillbits,
String defaultUserPort)
+throws InvalidConnectionInfoException {
+// If no drillbits is provided then throw exception
+drillbits = drillbits.trim();
+if (drillbits.isEmpty()) {
+ throw new InvalidConnectionInfoException("No drillbit information
specified in the connection string");
+}
+
+ArrayList endpointList = new ArrayList<>();
--- End diff --
fixed
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
> Labels: ready-to-commit
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15706601#comment-15706601
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r90090906
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/client/DrillClientTest.java
---
@@ -0,0 +1,258 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.client;
+
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.exec.DrillSystemTestBase;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.rpc.InvalidConnectionInfoException;
+import org.junit.Test;
+import java.util.List;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+
+/**
+ * The unit test case will read a physical plan in json format. The
physical plan contains a "trace" operator,
+ * which will produce a dump file. The dump file will be input into
DumpCat to test query mode and batch mode.
+ */
+
+public class DrillClientTest extends DrillSystemTestBase {
+
+ private final DrillConfig config = DrillConfig.create();
+
+ @Test
+ public void testParseAndVerifyEndpointsSingleDrillbitIp() throws
Exception {
+
+// Test with single drillbit ip
+final String drillBitConnection = "10.10.100.161";
+final List endpointsList =
DrillClient.parseAndVerifyEndpoints
+ (drillBitConnection,
config.getString(ExecConstants.INITIAL_USER_PORT));
+final CoordinationProtos.DrillbitEndpoint endpoint =
endpointsList.get(0);
+assertEquals(endpointsList.size(), 1);
+assertEquals(endpoint.getAddress(), drillBitConnection);
+assertEquals(endpoint.getUserPort(),
config.getInt(ExecConstants.INITIAL_USER_PORT));
+ }
+
+ @Test
+ public void testParseAndVerifyEndpointsSingleDrillbitIpPort() throws
Exception {
+
+// Test with single drillbit ip:port
+final String drillBitConnection = "10.10.100.161:5000";
+final String[] ipAndPort = drillBitConnection.split(":");
+final List endpointsList =
DrillClient.parseAndVerifyEndpoints
+ (drillBitConnection,
config.getString(ExecConstants.INITIAL_USER_PORT));
+assertEquals(endpointsList.size(), 1);
+
+final CoordinationProtos.DrillbitEndpoint endpoint =
endpointsList.get(0);
+assertEquals(endpoint.getAddress(), ipAndPort[0]);
+assertEquals(endpoint.getUserPort(), Integer.parseInt(ipAndPort[1]));
+ }
+
+ @Test
+ public void testParseAndVerifyEndpointsMultipleDrillbitIp() throws
Exception {
+
+// Test with multiple drillbit ip
+final String drillBitConnection = "10.10.100.161,10.10.100.162";
+final List endpointsList =
DrillClient.parseAndVerifyEndpoints
+ (drillBitConnection,
config.getString(ExecConstants.INITIAL_USER_PORT));
+assertEquals(endpointsList.size(), 2);
+
+CoordinationProtos.DrillbitEndpoint endpoint = endpointsList.get(0);
+assertEquals(endpoint.getAddress(), "10.10.100.161");
+assertEquals(endpoint.getUserPort(),
config.getInt(ExecConstants.INITIAL_USER_PORT));
+
+endpoint = endpointsList.get(1);
+assertEquals(endpoint.getAddress(), "10.10.100.162");
+assertEquals(endpoint.getUserPort(),
config.getInt(ExecConstants.INITIAL_USER_PORT));
+ }
+
+ @Test
+ public void testParseAndVerifyEndpointsMultipleDrillbitIpPort() throws
Exception {
+
+// Test with multiple drillbit ip:port
+final String drillBitConnection =
"10.10.100.161:5000,10.10.100.162:5000";
+final List endpointsList =
DrillClient.parseAndVerifyEndpoints
+ (drillBitConnection,
config.getString(ExecConstants.INITIAL_USER_PORT));
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[ https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15706602#comment-15706602 ] ASF GitHub Bot commented on DRILL-5015: --- Github user sohami commented on a diff in the pull request: https://github.com/apache/drill/pull/648#discussion_r89928643 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/client/DrillClientSystemTest.java --- @@ -17,11 +17,15 @@ */ package org.apache.drill.exec.client; +import java.util.ArrayList; --- End diff -- I will remove this file from the list since have moved all the changes to different file "DrillClientTest.java" > As per documentation, when issuing a list of drillbits in the connection > string, we always attempt to connect only to the first one > --- > > Key: DRILL-5015 > URL: https://issues.apache.org/jira/browse/DRILL-5015 > Project: Apache Drill > Issue Type: Bug > Components: Client - JDBC >Affects Versions: 1.8.0, 1.9.0 >Reporter: Sorabh Hamirwasia >Assignee: Sudheesh Katkam > Labels: ready-to-commit > > When trying to connect to a Drill cluster by specifying more than 1 drillbits > to connect to, we always attempt to connect to only the first drillbit. > As an example, we tested against a pair of drillbits, but we always connect > to the first entry in the CSV list by querying for the 'current' drillbit. > The remaining entries are never attempted. > [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u > "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f > whereAmI.q | grep -v logback > 1/1 select * from sys.drillbits where `current`; > +-++---++--+ > |hostname | user_port | control_port | data_port | current | > +-++---++--+ > | pssc-61.qa.lab | 31010 | 31011 | 31012 | true | > +-++---++--+ > 1 row selected (0.265 seconds) > Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl > apache drill 1.8.0 > "a little sql for your nosql" > This property is meant for use by clients when not wanting to overload the ZK > for fetching a list of existing Drillbits, but the behaviour doesn't match > the documentation. > [Making a Direct Drillbit Connection | > https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection > ] > We need to randomly shuffle between this list and If an entry in the shuffled > list is unreachable, we need to try for the next entry in the list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15706600#comment-15706600
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r89929966
--- Diff:
exec/rpc/src/main/java/org/apache/drill/exec/rpc/InvalidConnectionInfoException.java
---
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.rpc;
+
+/**
+ * Exception for malformed connection string from client
+ */
+public class InvalidConnectionInfoException extends RpcException {
--- End diff --
The connect method only throws RPCException. Also didn't want to extend the
method signature to throw one more checked exception since overtime that can
grow.
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
> Labels: ready-to-commit
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15706597#comment-15706597
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r89928539
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -245,14 +327,15 @@ public synchronized void connect(String connect,
Properties props) throws RpcExc
throw new RpcException("Failure setting up ZK for client.", e);
}
}
-
- final ArrayList endpoints = new
ArrayList<>(clusterCoordinator.getAvailableEndpoints());
- checkState(!endpoints.isEmpty(), "No DrillbitEndpoint can be found");
- // shuffle the collection then get the first endpoint
- Collections.shuffle(endpoints);
- endpoint = endpoints.iterator().next();
+ endpoints.addAll(clusterCoordinator.getAvailableEndpoints());
+ // Make sure we have at least one endpoint in the list
+ checkState(!endpoints.isEmpty(), "No active Drillbit endpoint found
from zookeeper");
--- End diff --
"checkState" is only called when drillbit information is populated using
ZooKeeper. For connection string case we are throwing
InvalidConnectionInfoException. Changed the string to "ZooKeeper"
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
> Labels: ready-to-commit
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[ https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15706599#comment-15706599 ] ASF GitHub Bot commented on DRILL-5015: --- Github user sohami commented on a diff in the pull request: https://github.com/apache/drill/pull/648#discussion_r90091315 --- Diff: exec/rpc/src/main/java/org/apache/drill/exec/rpc/InvalidConnectionInfoException.java --- @@ -0,0 +1,29 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.drill.exec.rpc; --- End diff -- discussed offline to keep it as is. > As per documentation, when issuing a list of drillbits in the connection > string, we always attempt to connect only to the first one > --- > > Key: DRILL-5015 > URL: https://issues.apache.org/jira/browse/DRILL-5015 > Project: Apache Drill > Issue Type: Bug > Components: Client - JDBC >Affects Versions: 1.8.0, 1.9.0 >Reporter: Sorabh Hamirwasia >Assignee: Sudheesh Katkam > Labels: ready-to-commit > > When trying to connect to a Drill cluster by specifying more than 1 drillbits > to connect to, we always attempt to connect to only the first drillbit. > As an example, we tested against a pair of drillbits, but we always connect > to the first entry in the CSV list by querying for the 'current' drillbit. > The remaining entries are never attempted. > [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u > "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f > whereAmI.q | grep -v logback > 1/1 select * from sys.drillbits where `current`; > +-++---++--+ > |hostname | user_port | control_port | data_port | current | > +-++---++--+ > | pssc-61.qa.lab | 31010 | 31011 | 31012 | true | > +-++---++--+ > 1 row selected (0.265 seconds) > Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl > apache drill 1.8.0 > "a little sql for your nosql" > This property is meant for use by clients when not wanting to overload the ZK > for fetching a list of existing Drillbits, but the behaviour doesn't match > the documentation. > [Making a Direct Drillbit Connection | > https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection > ] > We need to randomly shuffle between this list and If an entry in the shuffled > list is unreachable, we need to try for the next entry in the list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15703382#comment-15703382
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sudheeshkatkam commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r89902384
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/client/DrillClientTest.java
---
@@ -0,0 +1,258 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.client;
+
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.exec.DrillSystemTestBase;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.rpc.InvalidConnectionInfoException;
+import org.junit.Test;
+import java.util.List;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+
+/**
+ * The unit test case will read a physical plan in json format. The
physical plan contains a "trace" operator,
+ * which will produce a dump file. The dump file will be input into
DumpCat to test query mode and batch mode.
+ */
+
+public class DrillClientTest extends DrillSystemTestBase {
+
+ private final DrillConfig config = DrillConfig.create();
+
+ @Test
+ public void testParseAndVerifyEndpointsSingleDrillbitIp() throws
Exception {
+
+// Test with single drillbit ip
+final String drillBitConnection = "10.10.100.161";
+final List endpointsList =
DrillClient.parseAndVerifyEndpoints
+ (drillBitConnection,
config.getString(ExecConstants.INITIAL_USER_PORT));
+final CoordinationProtos.DrillbitEndpoint endpoint =
endpointsList.get(0);
+assertEquals(endpointsList.size(), 1);
+assertEquals(endpoint.getAddress(), drillBitConnection);
+assertEquals(endpoint.getUserPort(),
config.getInt(ExecConstants.INITIAL_USER_PORT));
+ }
+
+ @Test
+ public void testParseAndVerifyEndpointsSingleDrillbitIpPort() throws
Exception {
+
+// Test with single drillbit ip:port
+final String drillBitConnection = "10.10.100.161:5000";
+final String[] ipAndPort = drillBitConnection.split(":");
+final List endpointsList =
DrillClient.parseAndVerifyEndpoints
+ (drillBitConnection,
config.getString(ExecConstants.INITIAL_USER_PORT));
+assertEquals(endpointsList.size(), 1);
+
+final CoordinationProtos.DrillbitEndpoint endpoint =
endpointsList.get(0);
+assertEquals(endpoint.getAddress(), ipAndPort[0]);
+assertEquals(endpoint.getUserPort(), Integer.parseInt(ipAndPort[1]));
+ }
+
+ @Test
+ public void testParseAndVerifyEndpointsMultipleDrillbitIp() throws
Exception {
+
+// Test with multiple drillbit ip
+final String drillBitConnection = "10.10.100.161,10.10.100.162";
+final List endpointsList =
DrillClient.parseAndVerifyEndpoints
+ (drillBitConnection,
config.getString(ExecConstants.INITIAL_USER_PORT));
+assertEquals(endpointsList.size(), 2);
+
+CoordinationProtos.DrillbitEndpoint endpoint = endpointsList.get(0);
+assertEquals(endpoint.getAddress(), "10.10.100.161");
+assertEquals(endpoint.getUserPort(),
config.getInt(ExecConstants.INITIAL_USER_PORT));
+
+endpoint = endpointsList.get(1);
+assertEquals(endpoint.getAddress(), "10.10.100.162");
+assertEquals(endpoint.getUserPort(),
config.getInt(ExecConstants.INITIAL_USER_PORT));
+ }
+
+ @Test
+ public void testParseAndVerifyEndpointsMultipleDrillbitIpPort() throws
Exception {
+
+// Test with multiple drillbit ip:port
+final String drillBitConnection =
"10.10.100.161:5000,10.10.100.162:5000";
+final List endpointsList =
DrillClient.parseAndVerifyEndpoints
+ (drillBitConnection,
config.getString(ExecConstants.INITIAL_USER
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[ https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15703387#comment-15703387 ] ASF GitHub Bot commented on DRILL-5015: --- Github user sudheeshkatkam commented on a diff in the pull request: https://github.com/apache/drill/pull/648#discussion_r89903250 --- Diff: exec/rpc/src/main/java/org/apache/drill/exec/rpc/InvalidConnectionInfoException.java --- @@ -0,0 +1,29 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.drill.exec.rpc; --- End diff -- Should this be in the client package? > As per documentation, when issuing a list of drillbits in the connection > string, we always attempt to connect only to the first one > --- > > Key: DRILL-5015 > URL: https://issues.apache.org/jira/browse/DRILL-5015 > Project: Apache Drill > Issue Type: Bug > Components: Client - JDBC >Affects Versions: 1.8.0, 1.9.0 >Reporter: Sorabh Hamirwasia >Assignee: Sudheesh Katkam > Labels: ready-to-commit > > When trying to connect to a Drill cluster by specifying more than 1 drillbits > to connect to, we always attempt to connect to only the first drillbit. > As an example, we tested against a pair of drillbits, but we always connect > to the first entry in the CSV list by querying for the 'current' drillbit. > The remaining entries are never attempted. > [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u > "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f > whereAmI.q | grep -v logback > 1/1 select * from sys.drillbits where `current`; > +-++---++--+ > |hostname | user_port | control_port | data_port | current | > +-++---++--+ > | pssc-61.qa.lab | 31010 | 31011 | 31012 | true | > +-++---++--+ > 1 row selected (0.265 seconds) > Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl > apache drill 1.8.0 > "a little sql for your nosql" > This property is meant for use by clients when not wanting to overload the ZK > for fetching a list of existing Drillbits, but the behaviour doesn't match > the documentation. > [Making a Direct Drillbit Connection | > https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection > ] > We need to randomly shuffle between this list and If an entry in the shuffled > list is unreachable, we need to try for the next entry in the list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15703389#comment-15703389
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sudheeshkatkam commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r89902255
--- Diff:
exec/rpc/src/main/java/org/apache/drill/exec/rpc/InvalidConnectionInfoException.java
---
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.rpc;
+
+/**
+ * Exception for malformed connection string from client
+ */
+public class InvalidConnectionInfoException extends RpcException {
--- End diff --
Why extend RpcException?
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
> Labels: ready-to-commit
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15703384#comment-15703384
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sudheeshkatkam commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r89902789
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +224,100 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Populates the endpointlist with drillbits information provided in the
connection string by client.
+ * For direct connection we can have connection string with drillbit
property as below:
+ *
+ * drillbit=ip
+ * use the ip specified as the Foreman ip with default port in
config file
+ * drillbit=ip:port
+ * use the ip and port specified as the Foreman ip and port
+ * drillbit=ip1:port1,ip2:port2,...
+ * randomly select the ip and port pair from the specified list as
the Foreman ip and port.
+ *
+ *
+ * @param drillbits string with drillbit value provided in connection
string
+ * @param defaultUserPort string with default userport of drillbit
specified in config file
+ * @return list of drillbit endpoints parsed from connection string
+ * @throws InvalidConnectionInfoException if the connection string has
invalid or no drillbit information
+ */
+ static List parseAndVerifyEndpoints(String drillbits,
String defaultUserPort)
+throws InvalidConnectionInfoException {
+// If no drillbits is provided then throw exception
+drillbits = drillbits.trim();
+if (drillbits.isEmpty()) {
+ throw new InvalidConnectionInfoException("No drillbit information
specified in the connection string");
+}
+
+ArrayList endpointList = new ArrayList<>();
+final String[] connectInfo = drillbits.split(",");
+
+// Fetch ip address and port information for each drillbit and
populate the list
+for (String drillbit : connectInfo) {
+
+ // Trim all the empty spaces and check if the entry is empty string.
+ // Ignore the empty ones.
+ drillbit = drillbit.trim();
+
+ if (!drillbit.isEmpty()) {
+// Verify if we have only ":" or only ":port" pattern
+if (drillbit.charAt(0) == ':') {
+ // Invalid drillbit information
+ throw new InvalidConnectionInfoException("Malformed connection
string with drillbit hostname or " +
+ "hostaddress missing
for an entry: " + drillbit);
+}
+
+// We are now sure that each ip:port entry will have both the
values atleast once.
+// Split each drillbit connection string to get ip address and
port value
+final String[] drillbitInfo = drillbit.split(":");
+
+// Check if we have more than one port
+if (drillbitInfo.length > 2) {
+ throw new InvalidConnectionInfoException("Malformed connection
string with more than one port in a " +
+ "drillbit entry: " +
drillbit);
+}
+
+// At this point we are sure that drillbitInfo has atleast
hostname or host address
+// trim all the empty spaces which might be present in front of
hostname or
+// host address information
+final String ipAddress = drillbitInfo[0].trim();
+String port = defaultUserPort;
+
+if (drillbitInfo.length == 2) {
+ // We have a port value also given by user. trim all the empty
spaces between : and port value before
+ // validating the correctness of value.
+ port = drillbitInfo[1].trim();
+}
+
+try {
+ final DrillbitEndpoint endpoint = DrillbitEndpoint.newBuilder()
+.setAddress(ipAddress)
+
.setUserPort(Integer.parseInt(port))
+.build();
+
+ endpointList.add(endpoint);
+} catch (NumberFormatException e) {
+ throw new InvalidConnectionInfoException("Malformed port value
in entry: " + ipAddress + ":" + port + " " +
+ "passed in connection
string");
+}
+ }
+}
+if(endpointList.size() == 0){
--- End diff --
spacing fix
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the fir
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15703388#comment-15703388
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sudheeshkatkam commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r89902545
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -245,14 +327,15 @@ public synchronized void connect(String connect,
Properties props) throws RpcExc
throw new RpcException("Failure setting up ZK for client.", e);
}
}
-
- final ArrayList endpoints = new
ArrayList<>(clusterCoordinator.getAvailableEndpoints());
- checkState(!endpoints.isEmpty(), "No DrillbitEndpoint can be found");
- // shuffle the collection then get the first endpoint
- Collections.shuffle(endpoints);
- endpoint = endpoints.iterator().next();
+ endpoints.addAll(clusterCoordinator.getAvailableEndpoints());
+ // Make sure we have at least one endpoint in the list
+ checkState(!endpoints.isEmpty(), "No active Drillbit endpoint found
from zookeeper");
--- End diff --
Not necessarily "from ZooKeeper"?
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
> Labels: ready-to-commit
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15703386#comment-15703386
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sudheeshkatkam commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r89902276
--- Diff:
exec/jdbc/src/main/java/org/apache/drill/jdbc/impl/DrillConnectionImpl.java ---
@@ -154,7 +155,9 @@ protected DrillConnectionImpl(DriverImpl driver,
AvaticaFactory factory,
this.client.connect(connect, info);
} catch (OutOfMemoryException e) {
throw new SQLException("Failure creating root allocator", e);
-} catch (RpcException e) {
+}catch (InvalidConnectionInfoException e){
--- End diff --
spacing fix
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
> Labels: ready-to-commit
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[ https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15703383#comment-15703383 ] ASF GitHub Bot commented on DRILL-5015: --- Github user sudheeshkatkam commented on a diff in the pull request: https://github.com/apache/drill/pull/648#discussion_r89902410 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/client/DrillClientSystemTest.java --- @@ -17,11 +17,15 @@ */ package org.apache.drill.exec.client; +import java.util.ArrayList; --- End diff -- unused imports > As per documentation, when issuing a list of drillbits in the connection > string, we always attempt to connect only to the first one > --- > > Key: DRILL-5015 > URL: https://issues.apache.org/jira/browse/DRILL-5015 > Project: Apache Drill > Issue Type: Bug > Components: Client - JDBC >Affects Versions: 1.8.0, 1.9.0 >Reporter: Sorabh Hamirwasia >Assignee: Sudheesh Katkam > Labels: ready-to-commit > > When trying to connect to a Drill cluster by specifying more than 1 drillbits > to connect to, we always attempt to connect to only the first drillbit. > As an example, we tested against a pair of drillbits, but we always connect > to the first entry in the CSV list by querying for the 'current' drillbit. > The remaining entries are never attempted. > [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u > "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f > whereAmI.q | grep -v logback > 1/1 select * from sys.drillbits where `current`; > +-++---++--+ > |hostname | user_port | control_port | data_port | current | > +-++---++--+ > | pssc-61.qa.lab | 31010 | 31011 | 31012 | true | > +-++---++--+ > 1 row selected (0.265 seconds) > Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl > apache drill 1.8.0 > "a little sql for your nosql" > This property is meant for use by clients when not wanting to overload the ZK > for fetching a list of existing Drillbits, but the behaviour doesn't match > the documentation. > [Making a Direct Drillbit Connection | > https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection > ] > We need to randomly shuffle between this list and If an entry in the shuffled > list is unreachable, we need to try for the next entry in the list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15703385#comment-15703385
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sudheeshkatkam commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r89902578
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +224,100 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Populates the endpointlist with drillbits information provided in the
connection string by client.
+ * For direct connection we can have connection string with drillbit
property as below:
+ *
+ * drillbit=ip
+ * use the ip specified as the Foreman ip with default port in
config file
+ * drillbit=ip:port
+ * use the ip and port specified as the Foreman ip and port
+ * drillbit=ip1:port1,ip2:port2,...
+ * randomly select the ip and port pair from the specified list as
the Foreman ip and port.
+ *
+ *
+ * @param drillbits string with drillbit value provided in connection
string
+ * @param defaultUserPort string with default userport of drillbit
specified in config file
+ * @return list of drillbit endpoints parsed from connection string
+ * @throws InvalidConnectionInfoException if the connection string has
invalid or no drillbit information
+ */
+ static List parseAndVerifyEndpoints(String drillbits,
String defaultUserPort)
+throws InvalidConnectionInfoException {
+// If no drillbits is provided then throw exception
+drillbits = drillbits.trim();
+if (drillbits.isEmpty()) {
+ throw new InvalidConnectionInfoException("No drillbit information
specified in the connection string");
+}
+
+ArrayList endpointList = new ArrayList<>();
--- End diff --
final List\ ...
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
> Labels: ready-to-commit
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[ https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15703359#comment-15703359 ] ASF GitHub Bot commented on DRILL-5015: --- Github user parthchandra commented on the issue: https://github.com/apache/drill/pull/648 +1 > As per documentation, when issuing a list of drillbits in the connection > string, we always attempt to connect only to the first one > --- > > Key: DRILL-5015 > URL: https://issues.apache.org/jira/browse/DRILL-5015 > Project: Apache Drill > Issue Type: Bug > Components: Client - JDBC >Affects Versions: 1.8.0, 1.9.0 >Reporter: Sorabh Hamirwasia >Assignee: Sudheesh Katkam > Labels: ready-to-commit > > When trying to connect to a Drill cluster by specifying more than 1 drillbits > to connect to, we always attempt to connect to only the first drillbit. > As an example, we tested against a pair of drillbits, but we always connect > to the first entry in the CSV list by querying for the 'current' drillbit. > The remaining entries are never attempted. > [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u > "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f > whereAmI.q | grep -v logback > 1/1 select * from sys.drillbits where `current`; > +-++---++--+ > |hostname | user_port | control_port | data_port | current | > +-++---++--+ > | pssc-61.qa.lab | 31010 | 31011 | 31012 | true | > +-++---++--+ > 1 row selected (0.265 seconds) > Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl > apache drill 1.8.0 > "a little sql for your nosql" > This property is meant for use by clients when not wanting to overload the ZK > for fetching a list of existing Drillbits, but the behaviour doesn't match > the documentation. > [Making a Direct Drillbit Connection | > https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection > ] > We need to randomly shuffle between this list and If an entry in the shuffled > list is unreachable, we need to try for the next entry in the list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15656486#comment-15656486
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87551806
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +224,94 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to Populate the endpointList with the list of drillbits
+ * provided in the connection string by client.
+ *
+ * For direct connection we can get URL string having drillbit property
as below:
+ * drillbit=: --- use the ip and port specified as the Foreman
ip and port
+ * drillbit=--- use the ip specified as the Foreman ip with
default port in config file
+ * drillbit=:,:... --- Randomly select the ip
and port pair from the specified list as the
+ * Foreman ip and port.
+ *
+ * @param drillbits string with drillbit value provided in connection
string
+ * @param defaultUserPort string with default userport of drillbit
specified in config file
+ * @return list of drillbitendpoints parsed from connection string
+ * @throws InvalidConnectionInfoException if the connection string has
invalid or no drillbit information
+ */
+ static List parseAndVerifyEndpoints(String drillbits,
String defaultUserPort)
+throws InvalidConnectionInfoException {
+// If no drillbits is provided then just return empty list.
+if (drillbits.trim().isEmpty()) {
+ throw new InvalidConnectionInfoException("No drillbit information
specified in the connection string");
+}
+
+ArrayList endpointList = new ArrayList<>();
+final String[] connectInfo = drillbits.split(",");
+
+// Fetch ip address and port information for each drillbit and
populate the list
+for (String drillbit : connectInfo) {
+
+ // Trim all the empty spaces and check if the entry is empty string.
+ // Ignore the empty ones.
+ drillbit = drillbit.trim();
+
+ if (!drillbit.isEmpty()) {
+// Verify if we have only ":" or only ":port" pattern
+if (drillbit.charAt(0) == ':') {
+ // Invalid drillbit information
+ throw new InvalidConnectionInfoException("Malformed connection
string with drillbit hostname or " +
+ "hostaddress missing
for an entry: " + drillbit);
+}
+
+// We are now sure that each ip:port entry will have some both the
entries.
+// Split each drillbit connection string to get ip address and
port value
+final String[] drillbitInfo = drillbit.split(":");
+
+// Check if we have more than one port
+if (drillbitInfo.length > 2) {
+ throw new InvalidConnectionInfoException("Malformed connection
string with more than one port in a " +
+ "drillbit entry: " +
drillbit);
+}
+
+// At this point we are sure that drillbitInfo has atleast
hostname or host address
+// trim all the empty spaces which might be present in front of
hostname or
+// host address information
+final String ipAddress = drillbitInfo[0].trim();
+String port = defaultUserPort;
+
+if (drillbitInfo.length == 2) {
+ // We have a port value also given by user. trim all the empty
spaces between : and port value before
+ // validating the correctness of value.
+ port = drillbitInfo[1].trim();
+}
+
+try {
+ final DrillbitEndpoint endpoint = DrillbitEndpoint.newBuilder()
+.setAddress(ipAddress)
+
.setUserPort(Integer.parseInt(port))
+.build();
+
+ endpointList.add(endpoint);
+} catch (NumberFormatException e) {
+ throw new InvalidConnectionInfoException("Malformed port value
in entry: " + ipAddress + ":" + port + " " +
+ "passed in connection
string");
+}
+ }
+}
+return endpointList;
--- End diff --
Nice Catch! Thanks! I forgot to trim the drillbits string and assign to
itself, rather I just checked in if condition. I have made the change. Still
when the string contains only ","
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15656485#comment-15656485
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87551816
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +224,94 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to Populate the endpointList with the list of drillbits
+ * provided in the connection string by client.
+ *
+ * For direct connection we can get URL string having drillbit property
as below:
--- End diff --
Thanks! Fixed.
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15656488#comment-15656488
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87551857
--- Diff:
exec/rpc/src/main/java/org/apache/drill/exec/rpc/InvalidConnectionInfoException.java
---
@@ -0,0 +1,35 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.rpc;
+
+/**
+ * Exception class to differentiate errors due to malformed connection
string from client
+ */
+public class InvalidConnectionInfoException extends RpcException {
+ static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(InvalidConnectionInfoException.class);
--- End diff --
Removed
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15656487#comment-15656487
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87551861
--- Diff:
exec/rpc/src/main/java/org/apache/drill/exec/rpc/InvalidConnectionInfoException.java
---
@@ -0,0 +1,35 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.rpc;
+
+/**
+ * Exception class to differentiate errors due to malformed connection
string from client
+ */
+public class InvalidConnectionInfoException extends RpcException {
+ static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(InvalidConnectionInfoException.class);
+
+ private final String message;
+
+ public InvalidConnectionInfoException(String message) {
--- End diff --
Done
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15656001#comment-15656001
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87531891
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +224,94 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to Populate the endpointList with the list of drillbits
+ * provided in the connection string by client.
+ *
+ * For direct connection we can get URL string having drillbit property
as below:
+ * drillbit=: --- use the ip and port specified as the Foreman
ip and port
+ * drillbit=--- use the ip specified as the Foreman ip with
default port in config file
+ * drillbit=:,:... --- Randomly select the ip
and port pair from the specified list as the
+ * Foreman ip and port.
+ *
+ * @param drillbits string with drillbit value provided in connection
string
+ * @param defaultUserPort string with default userport of drillbit
specified in config file
+ * @return list of drillbitendpoints parsed from connection string
+ * @throws InvalidConnectionInfoException if the connection string has
invalid or no drillbit information
+ */
+ static List parseAndVerifyEndpoints(String drillbits,
String defaultUserPort)
+throws InvalidConnectionInfoException {
+// If no drillbits is provided then just return empty list.
+if (drillbits.trim().isEmpty()) {
+ throw new InvalidConnectionInfoException("No drillbit information
specified in the connection string");
+}
+
+ArrayList endpointList = new ArrayList<>();
+final String[] connectInfo = drillbits.split(",");
+
+// Fetch ip address and port information for each drillbit and
populate the list
+for (String drillbit : connectInfo) {
+
+ // Trim all the empty spaces and check if the entry is empty string.
+ // Ignore the empty ones.
+ drillbit = drillbit.trim();
+
+ if (!drillbit.isEmpty()) {
+// Verify if we have only ":" or only ":port" pattern
+if (drillbit.charAt(0) == ':') {
+ // Invalid drillbit information
+ throw new InvalidConnectionInfoException("Malformed connection
string with drillbit hostname or " +
+ "hostaddress missing
for an entry: " + drillbit);
+}
+
+// We are now sure that each ip:port entry will have some both the
entries.
+// Split each drillbit connection string to get ip address and
port value
+final String[] drillbitInfo = drillbit.split(":");
+
+// Check if we have more than one port
+if (drillbitInfo.length > 2) {
+ throw new InvalidConnectionInfoException("Malformed connection
string with more than one port in a " +
+ "drillbit entry: " +
drillbit);
+}
+
+// At this point we are sure that drillbitInfo has atleast
hostname or host address
+// trim all the empty spaces which might be present in front of
hostname or
+// host address information
+final String ipAddress = drillbitInfo[0].trim();
+String port = defaultUserPort;
+
+if (drillbitInfo.length == 2) {
+ // We have a port value also given by user. trim all the empty
spaces between : and port value before
+ // validating the correctness of value.
+ port = drillbitInfo[1].trim();
+}
+
+try {
+ final DrillbitEndpoint endpoint = DrillbitEndpoint.newBuilder()
+.setAddress(ipAddress)
+
.setUserPort(Integer.parseInt(port))
+.build();
+
+ endpointList.add(endpoint);
+} catch (NumberFormatException e) {
+ throw new InvalidConnectionInfoException("Malformed port value
in entry: " + ipAddress + ":" + port + " " +
+ "passed in connection
string");
+}
+ }
+}
+return endpointList;
--- End diff --
One last check: must have at least one endpoint. (The code above skips an
entry if it is empty. If that was the only entry, the endpoint list might be
empty here.)
> As per doc
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15656000#comment-15656000
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87531981
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +224,94 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to Populate the endpointList with the list of drillbits
+ * provided in the connection string by client.
+ *
+ * For direct connection we can get URL string having drillbit property
as below:
--- End diff --
Nit: this is a Javadoc comment, so strings must be formatted using HTML.
One handy tool:
drillbit=...
us the ip and port...
In Eclipse, hovering over the method name shows the formatted Javadoc for a
quick check.
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655998#comment-15655998
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87532223
--- Diff:
exec/rpc/src/main/java/org/apache/drill/exec/rpc/InvalidConnectionInfoException.java
---
@@ -0,0 +1,35 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.rpc;
+
+/**
+ * Exception class to differentiate errors due to malformed connection
string from client
+ */
+public class InvalidConnectionInfoException extends RpcException {
+ static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(InvalidConnectionInfoException.class);
--- End diff --
logger not strictly needed here: we don't log anything.
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655999#comment-15655999
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87532217
--- Diff:
exec/rpc/src/main/java/org/apache/drill/exec/rpc/InvalidConnectionInfoException.java
---
@@ -0,0 +1,35 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.rpc;
+
+/**
+ * Exception class to differentiate errors due to malformed connection
string from client
+ */
+public class InvalidConnectionInfoException extends RpcException {
+ static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(InvalidConnectionInfoException.class);
+
+ private final String message;
+
+ public InvalidConnectionInfoException(String message) {
--- End diff --
Just use the normal facilities:
public Invalid...( String message ) { super( message ); }
...
e.getMessage( ) ...
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653519#comment-15653519
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87351696
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
+ * provided in the connection string by client
+ * @param endpointList - ArrayList of DrillbitEndpoints
+ * @param drillbits - One or more drillbit ip[:port] provided in
connection string
+ */
+ public void populateEndpointsList(ArrayList
endpointList, String drillbits){
+
+// If no information about drillbits is provided then just return
empty list.
+if(drillbits == null || drillbits.length() == 0){
+ return;
+}
+final String[] connectInfo = drillbits.split(",");
+
+/* For direct connection we can get URL string having drillbit
property as below:
+ drillbit=: --- Use the IP and port specified as the
Foreman IP and port
+ drillbit=--- Use the IP specified as the Foreman IP
with default port in config file
+ drillbit=:,:... --- Randomly select the
IP and port pair from the specified
+ list as the Foreman
IP and port.
+
+ Fetch ip address and port information for each drillbit and
populate the list
+*/
+for(String info : connectInfo){
+ info = info.trim();
+
+ if(info != null){
+// Split each info to get ip address and port value
+final String[] drillbitInfo = info.split(":");
+
+// Check for malformed ip:port string
+if(drillbitInfo == null || drillbitInfo.length == 0){
+ continue;
+}
+
+/* If port is present use that one else use the configured one
+ Assumptions: 1) IP Address provided in connection string is
valid
+2) Port without IP address is never specified.
+*/
+final String port = (drillbitInfo.length == 2) ? drillbitInfo[1] :
config.getString(ExecConstants.INITIAL_USER_PORT);
+final DrillbitEndpoint endpoint = DrillbitEndpoint.newBuilder()
+ .setAddress(drillbitInfo[0])
+
.setUserPort(Integer.parseInt(port))
+ .build();
+endpointList.add(endpoint);
+ }
+}
+ }
+
public synchronized void connect(String connect, Properties props)
throws RpcException {
if (connected) {
return;
}
final DrillbitEndpoint endpoint;
+final ArrayList endpoints = new ArrayList<>();
--- End diff --
fixed
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653523#comment-15653523
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87351913
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
+ * provided in the connection string by client
+ * @param endpointList - ArrayList of DrillbitEndpoints
+ * @param drillbits - One or more drillbit ip[:port] provided in
connection string
+ */
+ public void populateEndpointsList(ArrayList
endpointList, String drillbits){
+
+// If no information about drillbits is provided then just return
empty list.
--- End diff --
Fixed
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653522#comment-15653522
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87351928
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
+ * provided in the connection string by client
+ * @param endpointList - ArrayList of DrillbitEndpoints
+ * @param drillbits - One or more drillbit ip[:port] provided in
connection string
--- End diff --
Just to clarify the I am not changing the variable name "drillbit" used in
the connection string. So there won't be any doc impact. The variable
"drillbits" is used in the internal function which is parsing the string.
As discussed the documentation is already there
[here](http://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection)
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653520#comment-15653520
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87351935
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
--- End diff --
Done
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653510#comment-15653510
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87351682
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
+ * provided in the connection string by client
+ * @param endpointList - ArrayList of DrillbitEndpoints
+ * @param drillbits - One or more drillbit ip[:port] provided in
connection string
+ */
+ public void populateEndpointsList(ArrayList
endpointList, String drillbits){
+
+// If no information about drillbits is provided then just return
empty list.
+if(drillbits == null || drillbits.length() == 0){
+ return;
+}
+final String[] connectInfo = drillbits.split(",");
+
+/* For direct connection we can get URL string having drillbit
property as below:
+ drillbit=: --- Use the IP and port specified as the
Foreman IP and port
+ drillbit=--- Use the IP specified as the Foreman IP
with default port in config file
+ drillbit=:,:... --- Randomly select the
IP and port pair from the specified
+ list as the Foreman
IP and port.
+
+ Fetch ip address and port information for each drillbit and
populate the list
+*/
+for(String info : connectInfo){
+ info = info.trim();
+
+ if(info != null){
+// Split each info to get ip address and port value
+final String[] drillbitInfo = info.split(":");
+
+// Check for malformed ip:port string
+if(drillbitInfo == null || drillbitInfo.length == 0){
+ continue;
+}
+
+/* If port is present use that one else use the configured one
+ Assumptions: 1) IP Address provided in connection string is
valid
+2) Port without IP address is never specified.
+*/
+final String port = (drillbitInfo.length == 2) ? drillbitInfo[1] :
config.getString(ExecConstants.INITIAL_USER_PORT);
+final DrillbitEndpoint endpoint = DrillbitEndpoint.newBuilder()
+ .setAddress(drillbitInfo[0])
+
.setUserPort(Integer.parseInt(port))
+ .build();
+endpointList.add(endpoint);
+ }
+}
+ }
+
public synchronized void connect(String connect, Properties props)
throws RpcException {
if (connected) {
return;
}
final DrillbitEndpoint endpoint;
+final ArrayList endpoints = new ArrayList<>();
if (isDirectConnection) {
- final String[] connectInfo =
props.getProperty("drillbit").split(":");
- final String port =
connectInfo.length==2?connectInfo[1]:config.getString(ExecConstants.INITIAL_USER_PORT);
- endpoint = DrillbitEndpoint.newBuilder()
- .setAddress(connectInfo[0])
- .setUserPort(Integer.parseInt(port))
- .build();
+ // Populate the endpoints list with all the drillbit information
provided in the
+ // connection string
+ populateEndpointsList(endpoints,
props.getProperty("drillbit").trim());
--- End diff --
Fixed. Changed the method to parseAndVerifyEndpoints
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sq
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653518#comment-15653518
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87351891
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
+ * provided in the connection string by client
+ * @param endpointList - ArrayList of DrillbitEndpoints
+ * @param drillbits - One or more drillbit ip[:port] provided in
connection string
+ */
+ public void populateEndpointsList(ArrayList
endpointList, String drillbits){
+
+// If no information about drillbits is provided then just return
empty list.
+if(drillbits == null || drillbits.length() == 0){
+ return;
+}
+final String[] connectInfo = drillbits.split(",");
+
+/* For direct connection we can get URL string having drillbit
property as below:
+ drillbit=: --- Use the IP and port specified as the
Foreman IP and port
+ drillbit=--- Use the IP specified as the Foreman IP
with default port in config file
+ drillbit=:,:... --- Randomly select the
IP and port pair from the specified
+ list as the Foreman
IP and port.
+
+ Fetch ip address and port information for each drillbit and
populate the list
+*/
+for(String info : connectInfo){
+ info = info.trim();
+
+ if(info != null){
+// Split each info to get ip address and port value
+final String[] drillbitInfo = info.split(":");
+
+// Check for malformed ip:port string
+if(drillbitInfo == null || drillbitInfo.length == 0){
--- End diff --
Length can be 0 here when the string just contain ":"
In case of more than 2 ports right now I am falling back to default port.
But as discussed changed it to throw new exception
"InvalidConnectionInfoException"
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653521#comment-15653521
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87351905
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
+ * provided in the connection string by client
+ * @param endpointList - ArrayList of DrillbitEndpoints
+ * @param drillbits - One or more drillbit ip[:port] provided in
connection string
+ */
+ public void populateEndpointsList(ArrayList
endpointList, String drillbits){
+
+// If no information about drillbits is provided then just return
empty list.
+if(drillbits == null || drillbits.length() == 0){
+ return;
+}
+final String[] connectInfo = drillbits.split(",");
+
+/* For direct connection we can get URL string having drillbit
property as below:
--- End diff --
Done
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653509#comment-15653509
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87351650
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/client/DrillClientSystemTest.java
---
@@ -73,4 +77,90 @@ public void testSubmitPlanTwoNodes() throws Exception {
}
client.close();
}
+
+ @Test
+ public void testPopulateEndpointsList() throws Exception{
+
+ArrayList endpointsList = new ArrayList<>();
+String drillBitConnection;
+DrillClient client = new DrillClient();
+DrillbitEndpoint endpoint;
+Iterator endpointIterator;
+
+
+// Test with single drillbit ip
+drillBitConnection = "10.10.100.161";
+client.populateEndpointsList(endpointsList, drillBitConnection);
+endpoint = endpointsList.iterator().next();
+assert(endpointsList.size() == 1);
+assert(endpoint.getAddress().equalsIgnoreCase(drillBitConnection));
+assert(endpoint.getUserPort() ==
client.getConfig().getInt(ExecConstants.INITIAL_USER_PORT));
+
+// Test with single drillbit ip:port
+endpointsList.clear();
--- End diff --
Fixed
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653512#comment-15653512
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87351880
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
+ * provided in the connection string by client
+ * @param endpointList - ArrayList of DrillbitEndpoints
+ * @param drillbits - One or more drillbit ip[:port] provided in
connection string
+ */
+ public void populateEndpointsList(ArrayList
endpointList, String drillbits){
+
+// If no information about drillbits is provided then just return
empty list.
+if(drillbits == null || drillbits.length() == 0){
+ return;
+}
+final String[] connectInfo = drillbits.split(",");
+
+/* For direct connection we can get URL string having drillbit
property as below:
+ drillbit=: --- Use the IP and port specified as the
Foreman IP and port
+ drillbit=--- Use the IP specified as the Foreman IP
with default port in config file
+ drillbit=:,:... --- Randomly select the
IP and port pair from the specified
+ list as the Foreman
IP and port.
+
+ Fetch ip address and port information for each drillbit and
populate the list
+*/
+for(String info : connectInfo){
+ info = info.trim();
+
+ if(info != null){
+// Split each info to get ip address and port value
+final String[] drillbitInfo = info.split(":");
+
+// Check for malformed ip:port string
+if(drillbitInfo == null || drillbitInfo.length == 0){
+ continue;
+}
+
+/* If port is present use that one else use the configured one
+ Assumptions: 1) IP Address provided in connection string is
valid
+2) Port without IP address is never specified.
+*/
+final String port = (drillbitInfo.length == 2) ? drillbitInfo[1] :
config.getString(ExecConstants.INITIAL_USER_PORT);
--- End diff --
Put the sanity checks in place for all error condition and throwing
exception with well formed messages.
For length > 2 case, treating it as error and throwing exception
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable,
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653515#comment-15653515
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87351685
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
+ * provided in the connection string by client
+ * @param endpointList - ArrayList of DrillbitEndpoints
+ * @param drillbits - One or more drillbit ip[:port] provided in
connection string
+ */
+ public void populateEndpointsList(ArrayList
endpointList, String drillbits){
+
+// If no information about drillbits is provided then just return
empty list.
+if(drillbits == null || drillbits.length() == 0){
+ return;
+}
+final String[] connectInfo = drillbits.split(",");
+
+/* For direct connection we can get URL string having drillbit
property as below:
+ drillbit=: --- Use the IP and port specified as the
Foreman IP and port
+ drillbit=--- Use the IP specified as the Foreman IP
with default port in config file
+ drillbit=:,:... --- Randomly select the
IP and port pair from the specified
+ list as the Foreman
IP and port.
+
+ Fetch ip address and port information for each drillbit and
populate the list
+*/
+for(String info : connectInfo){
+ info = info.trim();
+
+ if(info != null){
+// Split each info to get ip address and port value
+final String[] drillbitInfo = info.split(":");
+
+// Check for malformed ip:port string
+if(drillbitInfo == null || drillbitInfo.length == 0){
+ continue;
+}
+
+/* If port is present use that one else use the configured one
+ Assumptions: 1) IP Address provided in connection string is
valid
+2) Port without IP address is never specified.
+*/
+final String port = (drillbitInfo.length == 2) ? drillbitInfo[1] :
config.getString(ExecConstants.INITIAL_USER_PORT);
+final DrillbitEndpoint endpoint = DrillbitEndpoint.newBuilder()
+ .setAddress(drillbitInfo[0])
+
.setUserPort(Integer.parseInt(port))
+ .build();
+endpointList.add(endpoint);
+ }
+}
+ }
+
public synchronized void connect(String connect, Properties props)
throws RpcException {
if (connected) {
return;
}
final DrillbitEndpoint endpoint;
+final ArrayList endpoints = new ArrayList<>();
if (isDirectConnection) {
- final String[] connectInfo =
props.getProperty("drillbit").split(":");
- final String port =
connectInfo.length==2?connectInfo[1]:config.getString(ExecConstants.INITIAL_USER_PORT);
- endpoint = DrillbitEndpoint.newBuilder()
- .setAddress(connectInfo[0])
- .setUserPort(Integer.parseInt(port))
- .build();
+ // Populate the endpoints list with all the drillbit information
provided in the
+ // connection string
+ populateEndpointsList(endpoints,
props.getProperty("drillbit").trim());
--- End diff --
if "drillbit" is unset in the connection string then this code path won't
be called at all. If "drillbit" string is specified in connection string then
the value is set to empty string when none exists. So .trim() will not cause
NPE. But moved it in callee.
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we a
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653514#comment-15653514
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87351921
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
+ * provided in the connection string by client
+ * @param endpointList - ArrayList of DrillbitEndpoints
+ * @param drillbits - One or more drillbit ip[:port] provided in
connection string
+ */
+ public void populateEndpointsList(ArrayList
endpointList, String drillbits){
--- End diff --
Fixed
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653511#comment-15653511
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87120245
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
+ * provided in the connection string by client
+ * @param endpointList - ArrayList of DrillbitEndpoints
+ * @param drillbits - One or more drillbit ip[:port] provided in
connection string
--- End diff --
@kkhatua - I am not sure if I followed fully. This is an internal method
parameter and is not changing the name of "drillbit" parameter in connection
string. So it should not have any doc impact.
@paul-rogers - I am not introducing anything new here. The
[documentation](http://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection)
already specifies the usage but the correct implementation was lacking. It
also says we only support one port (user port) as is the case with zk.
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653508#comment-15653508
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87351662
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -245,14 +291,15 @@ public synchronized void connect(String connect,
Properties props) throws RpcExc
throw new RpcException("Failure setting up ZK for client.", e);
}
}
-
- final ArrayList endpoints = new
ArrayList<>(clusterCoordinator.getAvailableEndpoints());
- checkState(!endpoints.isEmpty(), "No DrillbitEndpoint can be found");
- // shuffle the collection then get the first endpoint
- Collections.shuffle(endpoints);
- endpoint = endpoints.iterator().next();
+ endpoints.addAll(clusterCoordinator.getAvailableEndpoints());
}
+// Make sure we have at least one endpoint in the list
+checkState(!endpoints.isEmpty(), "No DrillbitEndpoint can be found");
+// shuffle the collection then get the first endpoint
+Collections.shuffle(endpoints);
+endpoint = endpoints.iterator().next();
--- End diff --
Fixed
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653513#comment-15653513
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87351874
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
+ * provided in the connection string by client
+ * @param endpointList - ArrayList of DrillbitEndpoints
+ * @param drillbits - One or more drillbit ip[:port] provided in
connection string
+ */
+ public void populateEndpointsList(ArrayList
endpointList, String drillbits){
+
+// If no information about drillbits is provided then just return
empty list.
+if(drillbits == null || drillbits.length() == 0){
+ return;
+}
+final String[] connectInfo = drillbits.split(",");
+
+/* For direct connection we can get URL string having drillbit
property as below:
+ drillbit=: --- Use the IP and port specified as the
Foreman IP and port
+ drillbit=--- Use the IP specified as the Foreman IP
with default port in config file
+ drillbit=:,:... --- Randomly select the
IP and port pair from the specified
+ list as the Foreman
IP and port.
+
+ Fetch ip address and port information for each drillbit and
populate the list
+*/
+for(String info : connectInfo){
+ info = info.trim();
+
+ if(info != null){
+// Split each info to get ip address and port value
+final String[] drillbitInfo = info.split(":");
+
+// Check for malformed ip:port string
+if(drillbitInfo == null || drillbitInfo.length == 0){
+ continue;
+}
+
+/* If port is present use that one else use the configured one
+ Assumptions: 1) IP Address provided in connection string is
valid
+2) Port without IP address is never specified.
+*/
+final String port = (drillbitInfo.length == 2) ? drillbitInfo[1] :
config.getString(ExecConstants.INITIAL_USER_PORT);
+final DrillbitEndpoint endpoint = DrillbitEndpoint.newBuilder()
+ .setAddress(drillbitInfo[0])
+
.setUserPort(Integer.parseInt(port))
--- End diff --
I am catching the exception that can arise from parseInt and throwing it
back to the user as InvalidConnectionInfoException with proper error message.
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https:
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653506#comment-15653506
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87351621
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/client/DrillClientSystemTest.java
---
@@ -73,4 +77,90 @@ public void testSubmitPlanTwoNodes() throws Exception {
}
client.close();
}
+
+ @Test
+ public void testPopulateEndpointsList() throws Exception{
+
+ArrayList endpointsList = new ArrayList<>();
+String drillBitConnection;
+DrillClient client = new DrillClient();
+DrillbitEndpoint endpoint;
+Iterator endpointIterator;
+
+
+// Test with single drillbit ip
+drillBitConnection = "10.10.100.161";
+client.populateEndpointsList(endpointsList, drillBitConnection);
+endpoint = endpointsList.iterator().next();
+assert(endpointsList.size() == 1);
+assert(endpoint.getAddress().equalsIgnoreCase(drillBitConnection));
+assert(endpoint.getUserPort() ==
client.getConfig().getInt(ExecConstants.INITIAL_USER_PORT));
+
+// Test with single drillbit ip:port
+endpointsList.clear();
+drillBitConnection = "10.10.100.161:5000";
+String[] ipAndPort = drillBitConnection.split(":");
+client.populateEndpointsList(endpointsList, drillBitConnection);
+assert(endpointsList.size() == 1);
+
+endpoint = endpointsList.iterator().next();
+assert(endpoint.getAddress().equalsIgnoreCase(ipAndPort[0]));
+assert(endpoint.getUserPort() == Integer.parseInt(ipAndPort[1]));
+
+// Test with multiple drillbit ip
+endpointsList.clear();
+drillBitConnection = "10.10.100.161,10.10.100.162";
+client.populateEndpointsList(endpointsList, drillBitConnection);
+assert(endpointsList.size() == 2);
+
+endpointIterator = endpointsList.iterator();
+endpoint = endpointIterator.next();
+assert(endpoint.getAddress().equalsIgnoreCase("10.10.100.161"));
+assert(endpoint.getUserPort() ==
client.getConfig().getInt(ExecConstants.INITIAL_USER_PORT));
+
+endpoint = endpointIterator.next();
+assert(endpoint.getAddress().equalsIgnoreCase("10.10.100.162"));
+assert(endpoint.getUserPort() ==
client.getConfig().getInt(ExecConstants.INITIAL_USER_PORT));
+
+// Test with multiple drillbit ip:port
+endpointsList.clear();
+drillBitConnection = "10.10.100.161:5000,10.10.100.162:5000";
+client.populateEndpointsList(endpointsList, drillBitConnection);
+assert(endpointsList.size() == 2);
+
+endpointIterator = endpointsList.iterator();
+endpoint = endpointIterator.next();
+assert(endpoint.getAddress().equalsIgnoreCase("10.10.100.161"));
+assert(endpoint.getUserPort() == 5000);
+
+endpoint = endpointIterator.next();
+assert(endpoint.getAddress().equalsIgnoreCase("10.10.100.162"));
+assert(endpoint.getUserPort() == 5000);
+
+// Test with multiple drillbit with mix of ip:port and ip
+endpointsList.clear();
+drillBitConnection = "10.10.100.161:5000,10.10.100.162";
+client.populateEndpointsList(endpointsList, drillBitConnection);
+assert(endpointsList.size() == 2);
+
+endpointIterator = endpointsList.iterator();
+endpoint = endpointIterator.next();
+assert(endpoint.getAddress().equalsIgnoreCase("10.10.100.161"));
+assert(endpoint.getUserPort() == 5000);
+
+endpoint = endpointIterator.next();
+assert(endpoint.getAddress().equalsIgnoreCase("10.10.100.162"));
+assert(endpoint.getUserPort() ==
client.getConfig().getInt(ExecConstants.INITIAL_USER_PORT));
+
+// Test with empty string
+endpointsList.clear();
+drillBitConnection = "";
+client.populateEndpointsList(endpointsList, drillBitConnection);
+assert(endpointsList.size() == 0);
+
+
--- End diff --
Added more test case based on new implementation
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
>
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653505#comment-15653505
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87351670
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -245,14 +291,15 @@ public synchronized void connect(String connect,
Properties props) throws RpcExc
throw new RpcException("Failure setting up ZK for client.", e);
}
}
-
- final ArrayList endpoints = new
ArrayList<>(clusterCoordinator.getAvailableEndpoints());
- checkState(!endpoints.isEmpty(), "No DrillbitEndpoint can be found");
- // shuffle the collection then get the first endpoint
- Collections.shuffle(endpoints);
- endpoint = endpoints.iterator().next();
+ endpoints.addAll(clusterCoordinator.getAvailableEndpoints());
}
+// Make sure we have at least one endpoint in the list
+checkState(!endpoints.isEmpty(), "No DrillbitEndpoint can be found");
--- End diff --
Moved checkstate inside the else condition and change the string to capture
more information. For if case we are throwing exception if there is no
"drillbit" value passed in connection string
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653517#comment-15653517
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87351643
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/client/DrillClientSystemTest.java
---
@@ -73,4 +77,90 @@ public void testSubmitPlanTwoNodes() throws Exception {
}
client.close();
}
+
+ @Test
+ public void testPopulateEndpointsList() throws Exception{
+
+ArrayList endpointsList = new ArrayList<>();
+String drillBitConnection;
+DrillClient client = new DrillClient();
+DrillbitEndpoint endpoint;
+Iterator endpointIterator;
+
+
+// Test with single drillbit ip
+drillBitConnection = "10.10.100.161";
+client.populateEndpointsList(endpointsList, drillBitConnection);
+endpoint = endpointsList.iterator().next();
+assert(endpointsList.size() == 1);
+assert(endpoint.getAddress().equalsIgnoreCase(drillBitConnection));
+assert(endpoint.getUserPort() ==
client.getConfig().getInt(ExecConstants.INITIAL_USER_PORT));
+
+// Test with single drillbit ip:port
+endpointsList.clear();
+drillBitConnection = "10.10.100.161:5000";
+String[] ipAndPort = drillBitConnection.split(":");
+client.populateEndpointsList(endpointsList, drillBitConnection);
+assert(endpointsList.size() == 1);
+
+endpoint = endpointsList.iterator().next();
+assert(endpoint.getAddress().equalsIgnoreCase(ipAndPort[0]));
+assert(endpoint.getUserPort() == Integer.parseInt(ipAndPort[1]));
+
+// Test with multiple drillbit ip
+endpointsList.clear();
+drillBitConnection = "10.10.100.161,10.10.100.162";
+client.populateEndpointsList(endpointsList, drillBitConnection);
+assert(endpointsList.size() == 2);
+
+endpointIterator = endpointsList.iterator();
+endpoint = endpointIterator.next();
+assert(endpoint.getAddress().equalsIgnoreCase("10.10.100.161"));
+assert(endpoint.getUserPort() ==
client.getConfig().getInt(ExecConstants.INITIAL_USER_PORT));
--- End diff --
Fixed
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653507#comment-15653507
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87351895
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
+ * provided in the connection string by client
+ * @param endpointList - ArrayList of DrillbitEndpoints
+ * @param drillbits - One or more drillbit ip[:port] provided in
connection string
+ */
+ public void populateEndpointsList(ArrayList
endpointList, String drillbits){
+
+// If no information about drillbits is provided then just return
empty list.
+if(drillbits == null || drillbits.length() == 0){
+ return;
+}
+final String[] connectInfo = drillbits.split(",");
+
+/* For direct connection we can get URL string having drillbit
property as below:
+ drillbit=: --- Use the IP and port specified as the
Foreman IP and port
+ drillbit=--- Use the IP specified as the Foreman IP
with default port in config file
+ drillbit=:,:... --- Randomly select the
IP and port pair from the specified
+ list as the Foreman
IP and port.
+
+ Fetch ip address and port information for each drillbit and
populate the list
+*/
+for(String info : connectInfo){
+ info = info.trim();
+
+ if(info != null){
--- End diff --
Yes. Fixed. Changed to drillbit
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653516#comment-15653516
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user sohami commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87351912
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
+ * provided in the connection string by client
+ * @param endpointList - ArrayList of DrillbitEndpoints
+ * @param drillbits - One or more drillbit ip[:port] provided in
connection string
+ */
+ public void populateEndpointsList(ArrayList
endpointList, String drillbits){
+
+// If no information about drillbits is provided then just return
empty list.
+if(drillbits == null || drillbits.length() == 0){
--- End diff --
Fixed
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649185#comment-15649185
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87107869
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
+ * provided in the connection string by client
+ * @param endpointList - ArrayList of DrillbitEndpoints
+ * @param drillbits - One or more drillbit ip[:port] provided in
connection string
--- End diff --
Should keep the current name. Agree that "drillbit" should be documented
has having the same form as zk:
drillbit=node1[:port][,node2[:port]]*
The new form is backward compatible with the old form.
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649081#comment-15649081
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87100814
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -245,14 +291,15 @@ public synchronized void connect(String connect,
Properties props) throws RpcExc
throw new RpcException("Failure setting up ZK for client.", e);
}
}
-
- final ArrayList endpoints = new
ArrayList<>(clusterCoordinator.getAvailableEndpoints());
- checkState(!endpoints.isEmpty(), "No DrillbitEndpoint can be found");
- // shuffle the collection then get the first endpoint
- Collections.shuffle(endpoints);
- endpoint = endpoints.iterator().next();
+ endpoints.addAll(clusterCoordinator.getAvailableEndpoints());
}
+// Make sure we have at least one endpoint in the list
+checkState(!endpoints.isEmpty(), "No DrillbitEndpoint can be found");
+// shuffle the collection then get the first endpoint
+Collections.shuffle(endpoints);
+endpoint = endpoints.iterator().next();
--- End diff --
endpoint = endpoints.get( 0 );
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649071#comment-15649071
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87101443
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/client/DrillClientSystemTest.java
---
@@ -73,4 +77,90 @@ public void testSubmitPlanTwoNodes() throws Exception {
}
client.close();
}
+
+ @Test
+ public void testPopulateEndpointsList() throws Exception{
+
+ArrayList endpointsList = new ArrayList<>();
+String drillBitConnection;
+DrillClient client = new DrillClient();
+DrillbitEndpoint endpoint;
+Iterator endpointIterator;
+
+
+// Test with single drillbit ip
+drillBitConnection = "10.10.100.161";
+client.populateEndpointsList(endpointsList, drillBitConnection);
+endpoint = endpointsList.iterator().next();
+assert(endpointsList.size() == 1);
+assert(endpoint.getAddress().equalsIgnoreCase(drillBitConnection));
+assert(endpoint.getUserPort() ==
client.getConfig().getInt(ExecConstants.INITIAL_USER_PORT));
+
+// Test with single drillbit ip:port
+endpointsList.clear();
+drillBitConnection = "10.10.100.161:5000";
+String[] ipAndPort = drillBitConnection.split(":");
+client.populateEndpointsList(endpointsList, drillBitConnection);
+assert(endpointsList.size() == 1);
+
+endpoint = endpointsList.iterator().next();
+assert(endpoint.getAddress().equalsIgnoreCase(ipAndPort[0]));
+assert(endpoint.getUserPort() == Integer.parseInt(ipAndPort[1]));
+
+// Test with multiple drillbit ip
+endpointsList.clear();
+drillBitConnection = "10.10.100.161,10.10.100.162";
+client.populateEndpointsList(endpointsList, drillBitConnection);
+assert(endpointsList.size() == 2);
+
+endpointIterator = endpointsList.iterator();
+endpoint = endpointIterator.next();
+assert(endpoint.getAddress().equalsIgnoreCase("10.10.100.161"));
+assert(endpoint.getUserPort() ==
client.getConfig().getInt(ExecConstants.INITIAL_USER_PORT));
--- End diff --
Common practice:
assertEquals( expected, actual )
So:
assert(client.getConfig().getInt(ExecConstants.INITIAL_USER_PORT),
endpoint.getUserPort());
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649079#comment-15649079
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87099827
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
+ * provided in the connection string by client
+ * @param endpointList - ArrayList of DrillbitEndpoints
+ * @param drillbits - One or more drillbit ip[:port] provided in
connection string
+ */
+ public void populateEndpointsList(ArrayList
endpointList, String drillbits){
+
+// If no information about drillbits is provided then just return
empty list.
+if(drillbits == null || drillbits.length() == 0){
+ return;
+}
+final String[] connectInfo = drillbits.split(",");
+
+/* For direct connection we can get URL string having drillbit
property as below:
+ drillbit=: --- Use the IP and port specified as the
Foreman IP and port
+ drillbit=--- Use the IP specified as the Foreman IP
with default port in config file
+ drillbit=:,:... --- Randomly select the
IP and port pair from the specified
+ list as the Foreman
IP and port.
+
+ Fetch ip address and port information for each drillbit and
populate the list
+*/
+for(String info : connectInfo){
+ info = info.trim();
+
+ if(info != null){
+// Split each info to get ip address and port value
+final String[] drillbitInfo = info.split(":");
+
+// Check for malformed ip:port string
+if(drillbitInfo == null || drillbitInfo.length == 0){
+ continue;
+}
+
+/* If port is present use that one else use the configured one
+ Assumptions: 1) IP Address provided in connection string is
valid
+2) Port without IP address is never specified.
+*/
+final String port = (drillbitInfo.length == 2) ? drillbitInfo[1] :
config.getString(ExecConstants.INITIAL_USER_PORT);
--- End diff --
What about "host:" The array will be ["host",""].
And, what about "host: " (space after colon) or "host : 20" (space around
colon) or "host:20 " (space after port)?
May need a condition that is length == 2 & ! drillbitInfo[1].trim(
).isEmpty( ).
Also, what happens if length > 2?
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649080#comment-15649080
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87100146
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
+ * provided in the connection string by client
+ * @param endpointList - ArrayList of DrillbitEndpoints
+ * @param drillbits - One or more drillbit ip[:port] provided in
connection string
+ */
+ public void populateEndpointsList(ArrayList
endpointList, String drillbits){
+
+// If no information about drillbits is provided then just return
empty list.
+if(drillbits == null || drillbits.length() == 0){
+ return;
+}
+final String[] connectInfo = drillbits.split(",");
+
+/* For direct connection we can get URL string having drillbit
property as below:
+ drillbit=: --- Use the IP and port specified as the
Foreman IP and port
+ drillbit=--- Use the IP specified as the Foreman IP
with default port in config file
+ drillbit=:,:... --- Randomly select the
IP and port pair from the specified
+ list as the Foreman
IP and port.
+
+ Fetch ip address and port information for each drillbit and
populate the list
+*/
+for(String info : connectInfo){
+ info = info.trim();
+
+ if(info != null){
+// Split each info to get ip address and port value
+final String[] drillbitInfo = info.split(":");
+
+// Check for malformed ip:port string
+if(drillbitInfo == null || drillbitInfo.length == 0){
+ continue;
+}
+
+/* If port is present use that one else use the configured one
+ Assumptions: 1) IP Address provided in connection string is
valid
+2) Port without IP address is never specified.
+*/
+final String port = (drillbitInfo.length == 2) ? drillbitInfo[1] :
config.getString(ExecConstants.INITIAL_USER_PORT);
+final DrillbitEndpoint endpoint = DrillbitEndpoint.newBuilder()
+ .setAddress(drillbitInfo[0])
+
.setUserPort(Integer.parseInt(port))
+ .build();
+endpointList.add(endpoint);
+ }
+}
+ }
+
public synchronized void connect(String connect, Properties props)
throws RpcException {
if (connected) {
return;
}
final DrillbitEndpoint endpoint;
+final ArrayList endpoints = new ArrayList<>();
--- End diff --
final List = ...
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649072#comment-15649072
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87097059
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
--- End diff --
Thanks for the Javadoc!
Function to populate -> Populate
information of all the -> the list of
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649068#comment-15649068
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87098173
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
+ * provided in the connection string by client
+ * @param endpointList - ArrayList of DrillbitEndpoints
+ * @param drillbits - One or more drillbit ip[:port] provided in
connection string
--- End diff --
One or more drillbit -> drillbits
(Javadoc practice is to use lower case in param comments. See:
http://www.oracle.com/technetwork/java/javase/documentation/index-137868.html#styleguide)
Please copy detailed description from method body to here so it shows up in
Javadoc.
I one port allowed? Or, should the user specify all three? If only one,
what port numbers are used for the other two? If this is documented elsewhere,
just point to that location.
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649065#comment-15649065
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87099094
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
+ * provided in the connection string by client
+ * @param endpointList - ArrayList of DrillbitEndpoints
+ * @param drillbits - One or more drillbit ip[:port] provided in
connection string
+ */
+ public void populateEndpointsList(ArrayList
endpointList, String drillbits){
+
+// If no information about drillbits is provided then just return
empty list.
+if(drillbits == null || drillbits.length() == 0){
+ return;
+}
+final String[] connectInfo = drillbits.split(",");
+
+/* For direct connection we can get URL string having drillbit
property as below:
+ drillbit=: --- Use the IP and port specified as the
Foreman IP and port
+ drillbit=--- Use the IP specified as the Foreman IP
with default port in config file
+ drillbit=:,:... --- Randomly select the
IP and port pair from the specified
+ list as the Foreman
IP and port.
+
+ Fetch ip address and port information for each drillbit and
populate the list
+*/
+for(String info : connectInfo){
+ info = info.trim();
+
+ if(info != null){
+// Split each info to get ip address and port value
+final String[] drillbitInfo = info.split(":");
+
+// Check for malformed ip:port string
+if(drillbitInfo == null || drillbitInfo.length == 0){
--- End diff --
Will never be null. Length won't be 0 if the string is non-empty. But, we
checked the non-empty case. So, the length will be 1 or more. What if the
length is greater than 2 (drillbit:10:20:30)?
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649077#comment-15649077
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87098346
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
+ * provided in the connection string by client
+ * @param endpointList - ArrayList of DrillbitEndpoints
+ * @param drillbits - One or more drillbit ip[:port] provided in
connection string
+ */
+ public void populateEndpointsList(ArrayList
endpointList, String drillbits){
+
+// If no information about drillbits is provided then just return
empty list.
--- End diff --
no information about --> no drillbits
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649075#comment-15649075
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87102298
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/client/DrillClientSystemTest.java
---
@@ -73,4 +77,90 @@ public void testSubmitPlanTwoNodes() throws Exception {
}
client.close();
}
+
+ @Test
+ public void testPopulateEndpointsList() throws Exception{
+
+ArrayList endpointsList = new ArrayList<>();
+String drillBitConnection;
+DrillClient client = new DrillClient();
+DrillbitEndpoint endpoint;
+Iterator endpointIterator;
+
+
+// Test with single drillbit ip
+drillBitConnection = "10.10.100.161";
+client.populateEndpointsList(endpointsList, drillBitConnection);
+endpoint = endpointsList.iterator().next();
+assert(endpointsList.size() == 1);
+assert(endpoint.getAddress().equalsIgnoreCase(drillBitConnection));
+assert(endpoint.getUserPort() ==
client.getConfig().getInt(ExecConstants.INITIAL_USER_PORT));
+
+// Test with single drillbit ip:port
+endpointsList.clear();
+drillBitConnection = "10.10.100.161:5000";
+String[] ipAndPort = drillBitConnection.split(":");
+client.populateEndpointsList(endpointsList, drillBitConnection);
+assert(endpointsList.size() == 1);
+
+endpoint = endpointsList.iterator().next();
+assert(endpoint.getAddress().equalsIgnoreCase(ipAndPort[0]));
+assert(endpoint.getUserPort() == Integer.parseInt(ipAndPort[1]));
+
+// Test with multiple drillbit ip
+endpointsList.clear();
+drillBitConnection = "10.10.100.161,10.10.100.162";
+client.populateEndpointsList(endpointsList, drillBitConnection);
+assert(endpointsList.size() == 2);
+
+endpointIterator = endpointsList.iterator();
+endpoint = endpointIterator.next();
+assert(endpoint.getAddress().equalsIgnoreCase("10.10.100.161"));
+assert(endpoint.getUserPort() ==
client.getConfig().getInt(ExecConstants.INITIAL_USER_PORT));
+
+endpoint = endpointIterator.next();
+assert(endpoint.getAddress().equalsIgnoreCase("10.10.100.162"));
+assert(endpoint.getUserPort() ==
client.getConfig().getInt(ExecConstants.INITIAL_USER_PORT));
+
+// Test with multiple drillbit ip:port
+endpointsList.clear();
+drillBitConnection = "10.10.100.161:5000,10.10.100.162:5000";
+client.populateEndpointsList(endpointsList, drillBitConnection);
+assert(endpointsList.size() == 2);
+
+endpointIterator = endpointsList.iterator();
+endpoint = endpointIterator.next();
+assert(endpoint.getAddress().equalsIgnoreCase("10.10.100.161"));
+assert(endpoint.getUserPort() == 5000);
+
+endpoint = endpointIterator.next();
+assert(endpoint.getAddress().equalsIgnoreCase("10.10.100.162"));
+assert(endpoint.getUserPort() == 5000);
+
+// Test with multiple drillbit with mix of ip:port and ip
+endpointsList.clear();
+drillBitConnection = "10.10.100.161:5000,10.10.100.162";
+client.populateEndpointsList(endpointsList, drillBitConnection);
+assert(endpointsList.size() == 2);
+
+endpointIterator = endpointsList.iterator();
+endpoint = endpointIterator.next();
+assert(endpoint.getAddress().equalsIgnoreCase("10.10.100.161"));
+assert(endpoint.getUserPort() == 5000);
+
+endpoint = endpointIterator.next();
+assert(endpoint.getAddress().equalsIgnoreCase("10.10.100.162"));
+assert(endpoint.getUserPort() ==
client.getConfig().getInt(ExecConstants.INITIAL_USER_PORT));
+
+// Test with empty string
+endpointsList.clear();
+drillBitConnection = "";
+client.populateEndpointsList(endpointsList, drillBitConnection);
+assert(endpointsList.size() == 0);
+
+
--- End diff --
Test the other strange cases noted in earlier comments: spaces, non-numeric
ports, missing host, etc:
" foo : 10 ", "foo: ", "foo:", "foo:bar", ":20", etc.
for errors:
{
try { parseEndpoints( "foo:bogus" ); fail( ); }
catch( YourException e ) { }
}
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
>
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649073#comment-15649073
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87100068
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
+ * provided in the connection string by client
+ * @param endpointList - ArrayList of DrillbitEndpoints
+ * @param drillbits - One or more drillbit ip[:port] provided in
connection string
+ */
+ public void populateEndpointsList(ArrayList
endpointList, String drillbits){
+
+// If no information about drillbits is provided then just return
empty list.
+if(drillbits == null || drillbits.length() == 0){
+ return;
+}
+final String[] connectInfo = drillbits.split(",");
+
+/* For direct connection we can get URL string having drillbit
property as below:
+ drillbit=: --- Use the IP and port specified as the
Foreman IP and port
+ drillbit=--- Use the IP specified as the Foreman IP
with default port in config file
+ drillbit=:,:... --- Randomly select the
IP and port pair from the specified
+ list as the Foreman
IP and port.
+
+ Fetch ip address and port information for each drillbit and
populate the list
+*/
+for(String info : connectInfo){
+ info = info.trim();
+
+ if(info != null){
+// Split each info to get ip address and port value
+final String[] drillbitInfo = info.split(":");
+
+// Check for malformed ip:port string
+if(drillbitInfo == null || drillbitInfo.length == 0){
+ continue;
+}
+
+/* If port is present use that one else use the configured one
+ Assumptions: 1) IP Address provided in connection string is
valid
+2) Port without IP address is never specified.
+*/
+final String port = (drillbitInfo.length == 2) ? drillbitInfo[1] :
config.getString(ExecConstants.INITIAL_USER_PORT);
+final DrillbitEndpoint endpoint = DrillbitEndpoint.newBuilder()
+ .setAddress(drillbitInfo[0])
+
.setUserPort(Integer.parseInt(port))
--- End diff --
parseInt( ) will throw an (unchecked) exception if the port is not an
integer. That is, will throw an error for "host:foo".
This raises an interesting question, how are we doing error handling? Just
ignoring invalid entries? Should we throw a uniform exception instead?
InvalidArgumentException or something Drill-specific?
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by client
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649069#comment-15649069
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87101810
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/client/DrillClientSystemTest.java
---
@@ -73,4 +77,90 @@ public void testSubmitPlanTwoNodes() throws Exception {
}
client.close();
}
+
+ @Test
+ public void testPopulateEndpointsList() throws Exception{
+
+ArrayList endpointsList = new ArrayList<>();
+String drillBitConnection;
+DrillClient client = new DrillClient();
+DrillbitEndpoint endpoint;
+Iterator endpointIterator;
+
+
+// Test with single drillbit ip
+drillBitConnection = "10.10.100.161";
+client.populateEndpointsList(endpointsList, drillBitConnection);
+endpoint = endpointsList.iterator().next();
+assert(endpointsList.size() == 1);
+assert(endpoint.getAddress().equalsIgnoreCase(drillBitConnection));
+assert(endpoint.getUserPort() ==
client.getConfig().getInt(ExecConstants.INITIAL_USER_PORT));
+
+// Test with single drillbit ip:port
+endpointsList.clear();
--- End diff --
Having to remember `endpointsList.clear( )` every time is error-prone.
Better:
{
String drillBitConnection = "10.10.100.161";
List endpointList = parseEndpoints( drillbitConnection
);
... // Your tests here
}
The above creates a new block, and new case-specific variables, of each
specific test. Less chance of one test "bleeding through" into the next one.
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649078#comment-15649078
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87098478
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
+ * provided in the connection string by client
+ * @param endpointList - ArrayList of DrillbitEndpoints
+ * @param drillbits - One or more drillbit ip[:port] provided in
connection string
+ */
+ public void populateEndpointsList(ArrayList
endpointList, String drillbits){
+
+// If no information about drillbits is provided then just return
empty list.
+if(drillbits == null || drillbits.length() == 0){
--- End diff --
drillbits.length( ) == 0 --> drillbits.trim( ).isEmpty( )
The trim removes any bogus white space. IsEmpty( ) is a shorthand for
length( ) == 0
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649064#comment-15649064
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87100765
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -245,14 +291,15 @@ public synchronized void connect(String connect,
Properties props) throws RpcExc
throw new RpcException("Failure setting up ZK for client.", e);
}
}
-
- final ArrayList endpoints = new
ArrayList<>(clusterCoordinator.getAvailableEndpoints());
- checkState(!endpoints.isEmpty(), "No DrillbitEndpoint can be found");
- // shuffle the collection then get the first endpoint
- Collections.shuffle(endpoints);
- endpoint = endpoints.iterator().next();
+ endpoints.addAll(clusterCoordinator.getAvailableEndpoints());
}
+// Make sure we have at least one endpoint in the list
+checkState(!endpoints.isEmpty(), "No DrillbitEndpoint can be found");
--- End diff --
If this error is shown to the user, perhaps help them out a bit:
No Drillbit endpoints found in either the connect string
(drillbit=host[:port]) or in the drill-override.conf file.
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649076#comment-15649076
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87097440
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
+ * provided in the connection string by client
+ * @param endpointList - ArrayList of DrillbitEndpoints
+ * @param drillbits - One or more drillbit ip[:port] provided in
connection string
+ */
+ public void populateEndpointsList(ArrayList
endpointList, String drillbits){
--- End diff --
ArrayList -> List
Generally, create a specific list, but pass it around as the generic form.
(Allows changing the implementation without impacting clients.)
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649066#comment-15649066
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87098611
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
+ * provided in the connection string by client
+ * @param endpointList - ArrayList of DrillbitEndpoints
+ * @param drillbits - One or more drillbit ip[:port] provided in
connection string
+ */
+ public void populateEndpointsList(ArrayList
endpointList, String drillbits){
+
+// If no information about drillbits is provided then just return
empty list.
+if(drillbits == null || drillbits.length() == 0){
+ return;
+}
+final String[] connectInfo = drillbits.split(",");
+
+/* For direct connection we can get URL string having drillbit
property as below:
--- End diff --
Cool! Best to put this into the Javadoc comment for this function where it
can be seen in the Javadoc.
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649070#comment-15649070
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87100318
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
+ * provided in the connection string by client
+ * @param endpointList - ArrayList of DrillbitEndpoints
+ * @param drillbits - One or more drillbit ip[:port] provided in
connection string
+ */
+ public void populateEndpointsList(ArrayList
endpointList, String drillbits){
+
+// If no information about drillbits is provided then just return
empty list.
+if(drillbits == null || drillbits.length() == 0){
+ return;
+}
+final String[] connectInfo = drillbits.split(",");
+
+/* For direct connection we can get URL string having drillbit
property as below:
+ drillbit=: --- Use the IP and port specified as the
Foreman IP and port
+ drillbit=--- Use the IP specified as the Foreman IP
with default port in config file
+ drillbit=:,:... --- Randomly select the
IP and port pair from the specified
+ list as the Foreman
IP and port.
+
+ Fetch ip address and port information for each drillbit and
populate the list
+*/
+for(String info : connectInfo){
+ info = info.trim();
+
+ if(info != null){
+// Split each info to get ip address and port value
+final String[] drillbitInfo = info.split(":");
+
+// Check for malformed ip:port string
+if(drillbitInfo == null || drillbitInfo.length == 0){
+ continue;
+}
+
+/* If port is present use that one else use the configured one
+ Assumptions: 1) IP Address provided in connection string is
valid
+2) Port without IP address is never specified.
+*/
+final String port = (drillbitInfo.length == 2) ? drillbitInfo[1] :
config.getString(ExecConstants.INITIAL_USER_PORT);
+final DrillbitEndpoint endpoint = DrillbitEndpoint.newBuilder()
+ .setAddress(drillbitInfo[0])
+
.setUserPort(Integer.parseInt(port))
+ .build();
+endpointList.add(endpoint);
+ }
+}
+ }
+
public synchronized void connect(String connect, Properties props)
throws RpcException {
if (connected) {
return;
}
final DrillbitEndpoint endpoint;
+final ArrayList endpoints = new ArrayList<>();
if (isDirectConnection) {
- final String[] connectInfo =
props.getProperty("drillbit").split(":");
- final String port =
connectInfo.length==2?connectInfo[1]:config.getString(ExecConstants.INITIAL_USER_PORT);
- endpoint = DrillbitEndpoint.newBuilder()
- .setAddress(connectInfo[0])
- .setUserPort(Integer.parseInt(port))
- .build();
+ // Populate the endpoints list with all the drillbit information
provided in the
+ // connection string
+ populateEndpointsList(endpoints,
props.getProperty("drillbit").trim());
--- End diff --
What happens if "drillbit" is unset? Will the return from getProperty( ) be
null? If so, the ".trim( )" will throw an NPE. Solution: push the trim into the
method.
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649067#comment-15649067
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87100583
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
+ * provided in the connection string by client
+ * @param endpointList - ArrayList of DrillbitEndpoints
+ * @param drillbits - One or more drillbit ip[:port] provided in
connection string
+ */
+ public void populateEndpointsList(ArrayList
endpointList, String drillbits){
+
+// If no information about drillbits is provided then just return
empty list.
+if(drillbits == null || drillbits.length() == 0){
+ return;
+}
+final String[] connectInfo = drillbits.split(",");
+
+/* For direct connection we can get URL string having drillbit
property as below:
+ drillbit=: --- Use the IP and port specified as the
Foreman IP and port
+ drillbit=--- Use the IP specified as the Foreman IP
with default port in config file
+ drillbit=:,:... --- Randomly select the
IP and port pair from the specified
+ list as the Foreman
IP and port.
+
+ Fetch ip address and port information for each drillbit and
populate the list
+*/
+for(String info : connectInfo){
+ info = info.trim();
+
+ if(info != null){
+// Split each info to get ip address and port value
+final String[] drillbitInfo = info.split(":");
+
+// Check for malformed ip:port string
+if(drillbitInfo == null || drillbitInfo.length == 0){
+ continue;
+}
+
+/* If port is present use that one else use the configured one
+ Assumptions: 1) IP Address provided in connection string is
valid
+2) Port without IP address is never specified.
+*/
+final String port = (drillbitInfo.length == 2) ? drillbitInfo[1] :
config.getString(ExecConstants.INITIAL_USER_PORT);
+final DrillbitEndpoint endpoint = DrillbitEndpoint.newBuilder()
+ .setAddress(drillbitInfo[0])
+
.setUserPort(Integer.parseInt(port))
+ .build();
+endpointList.add(endpoint);
+ }
+}
+ }
+
public synchronized void connect(String connect, Properties props)
throws RpcException {
if (connected) {
return;
}
final DrillbitEndpoint endpoint;
+final ArrayList endpoints = new ArrayList<>();
if (isDirectConnection) {
- final String[] connectInfo =
props.getProperty("drillbit").split(":");
- final String port =
connectInfo.length==2?connectInfo[1]:config.getString(ExecConstants.INITIAL_USER_PORT);
- endpoint = DrillbitEndpoint.newBuilder()
- .setAddress(connectInfo[0])
- .setUserPort(Integer.parseInt(port))
- .build();
+ // Populate the endpoints list with all the drillbit information
provided in the
+ // connection string
+ populateEndpointsList(endpoints,
props.getProperty("drillbit").trim());
--- End diff --
Actually, the code here would be cleaner as:
final List endpoints = parseEndpoints(
props.getProperty("drillbit") );
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are neve
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649074#comment-15649074
]
ASF GitHub Bot commented on DRILL-5015:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/648#discussion_r87098807
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -223,19 +223,65 @@ public void connect(Properties props) throws
RpcException {
connect(null, props);
}
+ /**
+ * Function to populate the endpointList with information of all the
drillbits
+ * provided in the connection string by client
+ * @param endpointList - ArrayList of DrillbitEndpoints
+ * @param drillbits - One or more drillbit ip[:port] provided in
connection string
+ */
+ public void populateEndpointsList(ArrayList
endpointList, String drillbits){
+
+// If no information about drillbits is provided then just return
empty list.
+if(drillbits == null || drillbits.length() == 0){
+ return;
+}
+final String[] connectInfo = drillbits.split(",");
+
+/* For direct connection we can get URL string having drillbit
property as below:
+ drillbit=: --- Use the IP and port specified as the
Foreman IP and port
+ drillbit=--- Use the IP specified as the Foreman IP
with default port in config file
+ drillbit=:,:... --- Randomly select the
IP and port pair from the specified
+ list as the Foreman
IP and port.
+
+ Fetch ip address and port information for each drillbit and
populate the list
+*/
+for(String info : connectInfo){
+ info = info.trim();
+
+ if(info != null){
--- End diff --
info will never be null here. Did you mean info.isEmpty( )? That will catch
"foo,,bar".
info -> drillbit (The word info is a bit generic...)
> As per documentation, when issuing a list of drillbits in the connection
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect
> to the first entry in the CSV list by querying for the 'current' drillbit.
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u
> "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f
> whereAmI.q | grep -v logback
> 1/1 select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port | control_port | data_port | current |
> +-++---++--+
> | pssc-61.qa.lab | 31010 | 31011 | 31012 | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK
> for fetching a list of existing Drillbits, but the behaviour doesn't match
> the documentation.
> [Making a Direct Drillbit Connection |
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
> ]
> We need to randomly shuffle between this list and If an entry in the shuffled
> list is unreachable, we need to try for the next entry in the list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[ https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648973#comment-15648973 ] ASF GitHub Bot commented on DRILL-5015: --- Github user sohami commented on the issue: https://github.com/apache/drill/pull/648 @kkhatua - The unit test I have written is just for validating the newly introduced "populateEndpointsList" method. I am not actually connecting to the drillbit from within the test. > As per documentation, when issuing a list of drillbits in the connection > string, we always attempt to connect only to the first one > --- > > Key: DRILL-5015 > URL: https://issues.apache.org/jira/browse/DRILL-5015 > Project: Apache Drill > Issue Type: Bug > Components: Client - JDBC >Affects Versions: 1.8.0, 1.9.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia > > When trying to connect to a Drill cluster by specifying more than 1 drillbits > to connect to, we always attempt to connect to only the first drillbit. > As an example, we tested against a pair of drillbits, but we always connect > to the first entry in the CSV list by querying for the 'current' drillbit. > The remaining entries are never attempted. > [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u > "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f > whereAmI.q | grep -v logback > 1/1 select * from sys.drillbits where `current`; > +-++---++--+ > |hostname | user_port | control_port | data_port | current | > +-++---++--+ > | pssc-61.qa.lab | 31010 | 31011 | 31012 | true | > +-++---++--+ > 1 row selected (0.265 seconds) > Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl > apache drill 1.8.0 > "a little sql for your nosql" > This property is meant for use by clients when not wanting to overload the ZK > for fetching a list of existing Drillbits, but the behaviour doesn't match > the documentation. > [Making a Direct Drillbit Connection | > https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection > ] > We need to randomly shuffle between this list and If an entry in the shuffled > list is unreachable, we need to try for the next entry in the list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[ https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648954#comment-15648954 ] ASF GitHub Bot commented on DRILL-5015: --- GitHub user sohami opened a pull request: https://github.com/apache/drill/pull/648 DRILL-5015: Randomly select the drillbit from the list provided by user in connection string @sudheeshkatkam - Please help to review the changes You can merge this pull request into a Git repository by running: $ git pull https://github.com/sohami/drill DRILL-5015 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/648.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #648 commit e5ef1b62233b7f21868a094813d429a7b8afc5b1 Author: Sorabh Hamirwasia Date: 2016-11-08T19:27:57Z DRILL-5015: Randomly select the drillbit from the list provided by user in connection string > As per documentation, when issuing a list of drillbits in the connection > string, we always attempt to connect only to the first one > --- > > Key: DRILL-5015 > URL: https://issues.apache.org/jira/browse/DRILL-5015 > Project: Apache Drill > Issue Type: Bug > Components: Client - JDBC >Affects Versions: 1.8.0, 1.9.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia > > When trying to connect to a Drill cluster by specifying more than 1 drillbits > to connect to, we always attempt to connect to only the first drillbit. > As an example, we tested against a pair of drillbits, but we always connect > to the first entry in the CSV list by querying for the 'current' drillbit. > The remaining entries are never attempted. > [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u > "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f > whereAmI.q | grep -v logback > 1/1 select * from sys.drillbits where `current`; > +-++---++--+ > |hostname | user_port | control_port | data_port | current | > +-++---++--+ > | pssc-61.qa.lab | 31010 | 31011 | 31012 | true | > +-++---++--+ > 1 row selected (0.265 seconds) > Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl > apache drill 1.8.0 > "a little sql for your nosql" > This property is meant for use by clients when not wanting to overload the ZK > for fetching a list of existing Drillbits, but the behaviour doesn't match > the documentation. > [Making a Direct Drillbit Connection | > https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection > ] > We need to randomly shuffle between this list and If an entry in the shuffled > list is unreachable, we need to try for the next entry in the list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[ https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648916#comment-15648916 ] Parth Chandra commented on DRILL-5015: -- I wonder where that came from. Anyway, if we fix the code then we won't have to change the documentation :) > As per documentation, when issuing a list of drillbits in the connection > string, we always attempt to connect only to the first one > --- > > Key: DRILL-5015 > URL: https://issues.apache.org/jira/browse/DRILL-5015 > Project: Apache Drill > Issue Type: Bug > Components: Client - JDBC >Affects Versions: 1.8.0, 1.9.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia > > When trying to connect to a Drill cluster by specifying more than 1 drillbits > to connect to, we always attempt to connect to only the first drillbit. > As an example, we tested against a pair of drillbits, but we always connect > to the first entry in the CSV list by querying for the 'current' drillbit. > The remaining entries are never attempted. > [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u > "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f > whereAmI.q | grep -v logback > 1/1 select * from sys.drillbits where `current`; > +-++---++--+ > |hostname | user_port | control_port | data_port | current | > +-++---++--+ > | pssc-61.qa.lab | 31010 | 31011 | 31012 | true | > +-++---++--+ > 1 row selected (0.265 seconds) > Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl > apache drill 1.8.0 > "a little sql for your nosql" > This property is meant for use by clients when not wanting to overload the ZK > for fetching a list of existing Drillbits, but the behaviour doesn't match > the documentation. > [Making a Direct Drillbit Connection | > https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection > ] > We need to randomly shuffle between this list and If an entry in the shuffled > list is unreachable, we need to try for the next entry in the list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[ https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648909#comment-15648909 ] Sorabh Hamirwasia commented on DRILL-5015: -- Hi Parth, Sorry forgot to update the link to documentation page. At this [link | http://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection ] it does mention that we can provide a list of drillbits endpoints in the connection string. This will still preserve the idea of connecting to specific drillbit when we provide only 1 endpoint in the connection string. > As per documentation, when issuing a list of drillbits in the connection > string, we always attempt to connect only to the first one > --- > > Key: DRILL-5015 > URL: https://issues.apache.org/jira/browse/DRILL-5015 > Project: Apache Drill > Issue Type: Bug > Components: Client - JDBC >Affects Versions: 1.8.0, 1.9.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia > > When trying to connect to a Drill cluster by specifying more than 1 drillbits > to connect to, we always attempt to connect to only the first drillbit. > As an example, we tested against a pair of drillbits, but we always connect > to the first entry in the CSV list by querying for the 'current' drillbit. > The remaining entries are never attempted. > [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u > "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f > whereAmI.q | grep -v logback > 1/1 select * from sys.drillbits where `current`; > +-++---++--+ > |hostname | user_port | control_port | data_port | current | > +-++---++--+ > | pssc-61.qa.lab | 31010 | 31011 | 31012 | true | > +-++---++--+ > 1 row selected (0.265 seconds) > Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl > apache drill 1.8.0 > "a little sql for your nosql" > This property is meant for use by clients when not wanting to overload the ZK > for fetching a list of existing Drillbits, but the behaviour doesn't match > the documentation. > [Making a Direct Drillbit Connection | > https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection > ] > We need to randomly shuffle between this list and If an entry in the shuffled > list is unreachable, we need to try for the next entry in the list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one
[ https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648867#comment-15648867 ] Parth Chandra commented on DRILL-5015: -- I don't think we ever intended to support drillbit=endpoint1,endpoint2,...; the idea was to support connecting to a specific drillbit only. > As per documentation, when issuing a list of drillbits in the connection > string, we always attempt to connect only to the first one > --- > > Key: DRILL-5015 > URL: https://issues.apache.org/jira/browse/DRILL-5015 > Project: Apache Drill > Issue Type: Bug > Components: Client - JDBC >Affects Versions: 1.8.0, 1.9.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia > > When trying to connect to a Drill cluster by specifying more than 1 drillbits > to connect to, we always attempt to connect to only the first drillbit. > As an example, we tested against a pair of drillbits, but we always connect > to the first entry in the CSV list by querying for the 'current' drillbit. > The remaining entries are never attempted. > [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline -u > "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f > whereAmI.q | grep -v logback > 1/1 select * from sys.drillbits where `current`; > +-++---++--+ > |hostname | user_port | control_port | data_port | current | > +-++---++--+ > | pssc-61.qa.lab | 31010 | 31011 | 31012 | true | > +-++---++--+ > 1 row selected (0.265 seconds) > Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl > apache drill 1.8.0 > "a little sql for your nosql" > This property is meant for use by clients when not wanting to overload the ZK > for fetching a list of existing Drillbits, but the behaviour doesn't match > the documentation. > Making a Direct Drillbit Connection > We ned too raandomly shuffle between this list and If an entry in the > shuffled list is unreachable, we need to try for the next entry in the list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
