It isn't that simple. At least I couldn't make it work as expected. It
seems the jdbc fat jar has some dependencies on other drill/3rd
parties jars. Or am I missing something here?
Here are the jar files I use:
drill-jdbc-all-0.7.0-incubating-SNAPSHOT.jar
jline-1.0.jar
sqlline-1_0_2.jar
JAR_DIR=$HOME/sqlline/jars
$JAVA_HOME/bin/java -Dlog.path=/home/lih/sqlline/log/sqlline.log -ea
-cp
$JAR_DIR/sqlline-1_0_2.jar:$JAR_DIR/jline-1.0.jar:$JAR_DIR/drill-jdbc-all-0.7.0-incubating-SNAPSHOT.jar
\
sqlline.SqlLine -d org.apache.drill.jdbc.Driver --maxWidth=10000 -u
jdbc:drill:zk=stgace-launcher07.diy.corp.ne1.yahoo.com:2181/drill/drill-lih
It seems that it can make the connection correctly. However, if I run
a query, I don't get the result back
Connecting to
jdbc:drill:zk=stgace-launcher07.diy.corp.ne1.yahoo.com:2181/drill/drill-lih
22:22:33.097 [main] DEBUG o.a.drill.common.config.DrillConfig -
Loading configs at the following URLs
[jar:file:/home/lih/sqlline/jars/drill-jdbc-all-0.7.0-incubating-SNAPSHOT.jar!/drill-module.conf]
22:22:33.589 [main] DEBUG i.n.u.i.l.InternalLoggerFactory - Using
SLF4J as the default logging framework
22:22:33.602 [main] DEBUG i.n.util.internal.PlatformDependent0 -
java.nio.Buffer.address: available
22:22:33.603 [main] DEBUG i.n.util.internal.PlatformDependent0 -
sun.misc.Unsafe.theUnsafe: available
22:22:33.604 [main] DEBUG i.n.util.internal.PlatformDependent0 -
sun.misc.Unsafe.copyMemory: available
22:22:33.605 [main] DEBUG i.n.util.internal.PlatformDependent0 -
java.nio.Bits.unaligned: true
22:22:33.609 [main] DEBUG i.n.util.internal.PlatformDependent - UID: 21499
22:22:33.611 [main] DEBUG i.n.util.internal.PlatformDependent - Java version: 7
22:22:33.611 [main] DEBUG i.n.util.internal.PlatformDependent -
-Dio.netty.noUnsafe: false
22:22:33.611 [main] DEBUG i.n.util.internal.PlatformDependent -
sun.misc.Unsafe: available
22:22:33.612 [main] DEBUG i.n.util.internal.PlatformDependent -
-Dio.netty.noJavassist: false
22:22:33.780 [main] DEBUG i.n.util.internal.PlatformDependent -
Javassist: available
22:22:33.781 [main] DEBUG i.n.util.internal.PlatformDependent -
-Dio.netty.tmpdir: /tmp (java.io.tmpdir)
22:22:33.781 [main] DEBUG i.n.util.internal.PlatformDependent -
-Dio.netty.bitMode: 64 (sun.arch.data.model)
22:22:33.781 [main] DEBUG i.n.util.internal.PlatformDependent -
-Dio.netty.noPreferDirect: false
22:22:33.782 [main] DEBUG i.n.buffer.PooledByteBufAllocator -
-Dio.netty.allocator.numHeapArenas: 4
22:22:33.782 [main] DEBUG i.n.buffer.PooledByteBufAllocator -
-Dio.netty.allocator.numDirectArenas: 4
22:22:33.782 [main] DEBUG i.n.buffer.PooledByteBufAllocator -
-Dio.netty.allocator.pageSize: 8192
22:22:33.782 [main] DEBUG i.n.buffer.PooledByteBufAllocator -
-Dio.netty.allocator.maxOrder: 11
22:22:33.782 [main] DEBUG i.n.buffer.PooledByteBufAllocator -
-Dio.netty.allocator.chunkSize: 16777216
22:22:33.783 [main] DEBUG i.n.buffer.PooledByteBufAllocator -
-Dio.netty.allocator.tinyCacheSize: 512
22:22:33.783 [main] DEBUG i.n.buffer.PooledByteBufAllocator -
-Dio.netty.allocator.smallCacheSize: 256
22:22:33.783 [main] DEBUG i.n.buffer.PooledByteBufAllocator -
-Dio.netty.allocator.normalCacheSize: 64
22:22:33.783 [main] DEBUG i.n.buffer.PooledByteBufAllocator -
-Dio.netty.allocator.maxCachedBufferCapacity: 32768
22:22:33.783 [main] DEBUG i.n.buffer.PooledByteBufAllocator -
-Dio.netty.allocator.cacheTrimInterval: 8192
22:22:33.835 [main] DEBUG io.netty.util.ResourceLeakDetector -
-Dio.netty.leakDetectionLevel: simple
22:22:33.867 [main] DEBUG o.a.drill.common.config.DrillConfig -
Loading configs at the following URLs
[jar:file:/home/lih/sqlline/jars/drill-jdbc-all-0.7.0-incubating-SNAPSHOT.jar!/drill-module.conf]
22:22:33.938 [main] DEBUG org.reflections.Reflections - going to scan
these urls:
jar:file:/home/lih/sqlline/jars/drill-jdbc-all-0.7.0-incubating-SNAPSHOT.jar!/
22:22:38.791 [main] INFO org.reflections.Reflections - Reflections
took 4852 ms to scan 1 urls, producing 2602 keys and 9358 values
22:22:38.805 [main] DEBUG o.a.d.c.l.data.LogicalOperatorBase - Adding
Logical Operator sub types: [class
org.apache.drill.common.logical.data.Store, class
org.apache.drill.common.logical.data.GroupingAggregate, class
org.apache.drill.common.logical.data.Project, class
org.apache.drill.common.logical.data.Join, class
org.apache.drill.common.logical.data.Window, class
org.apache.drill.common.logical.data.Limit, class
org.apache.drill.common.logical.data.Filter, class
org.apache.drill.common.logical.data.Writer, class
org.apache.drill.common.logical.data.Scan, class
org.apache.drill.common.logical.data.Order, class
org.apache.drill.common.logical.data.Union]
22:22:38.811 [main] DEBUG o.a.d.c.l.StoragePluginConfigBase - Adding
Storage Engine Configs including [class
org.apache.drill.exec.store.NamedStoragePluginConfig, class
org.apache.drill.exec.store.sys.SystemTablePluginConfig, class
org.apache.drill.exec.store.dfs.FileSystemConfig, class
org.apache.drill.exec.store.ischema.InfoSchemaConfig]
22:22:38.814 [main] DEBUG o.a.d.c.l.FormatPluginConfigBase - Adding
Format Plugin Configs including [class
org.apache.drill.exec.store.dfs.NamedFormatPluginConfig]
22:22:38.820 [main] DEBUG o.a.d.e.c.zk.ZKClusterCoordinator - Connect
stgace-launcher07.diy.corp.ne1.yahoo.com:2181, zkRoot drill,
clusterId: drill-lih
22:22:38.885 [main] INFO o.a.c.f.imps.CuratorFrameworkImpl - Starting
22:22:38.889 [main] DEBUG o.a.curator.CuratorZookeeperClient - Starting
22:22:38.889 [main] DEBUG org.apache.curator.ConnectionState - Starting
22:22:38.889 [main] DEBUG org.apache.curator.ConnectionState - reset
22:22:38.903 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
GMT
22:22:38.903 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:host.name=stgace-launcher07.diy.corp.ne1.yahoo.com
22:22:38.903 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.version=1.7.0_17
22:22:38.903 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.vendor=Oracle Corporation
22:22:38.903 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.home=/home/y/share/gridjdk64-1.7.0_17/jre
22:22:38.903 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.class.path=/home/lih/sqlline/jars/sqlline-1_0_2.jar:/home/lih/sqlline/jars/jline-1.0.jar:/home/lih/sqlline/jars/drill-jdbc-all-0.7.0-incubating-SNAPSHOT.jar
22:22:38.903 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
22:22:38.903 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.io.tmpdir=/tmp
22:22:38.903 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.compiler=<NA>
22:22:38.903 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:os.name=Linux
22:22:38.904 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:os.arch=amd64
22:22:38.904 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:os.version=2.6.32-358.18.1.el6.YAHOO.20131009.x86_64
22:22:38.904 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:user.name=lih
22:22:38.904 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:user.home=/home/lih
22:22:38.904 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:user.dir=/home/lih/sqlline
22:22:38.906 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating
client connection,
connectString=stgace-launcher07.diy.corp.ne1.yahoo.com:2181
sessionTimeout=60000
watcher=org.apache.curator.ConnectionState@4bb526fb
22:22:38.913 [main] DEBUG org.apache.zookeeper.ClientCnxn -
zookeeper.disableAutoWatchReset is false
22:22:38.960 [main-SendThread(stgace-launcher07.diy.corp.ne1.yahoo.com:2181)]
INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to
server stgace-launcher07.diy.corp.ne1.yahoo.com/98.138.236.98:2181.
Will not attempt to authenticate using SASL (unknown error)
22:22:38.972 [main-SendThread(stgace-launcher07.diy.corp.ne1.yahoo.com:2181)]
INFO org.apache.zookeeper.ClientCnxn - Socket connection established
to stgace-launcher07.diy.corp.ne1.yahoo.com/98.138.236.98:2181,
initiating session
22:22:38.975 [main-SendThread(stgace-launcher07.diy.corp.ne1.yahoo.com:2181)]
DEBUG org.apache.zookeeper.ClientCnxn - Session establishment request
sent on stgace-launcher07.diy.corp.ne1.yahoo.com/98.138.236.98:2181
22:22:38.998 [main-SendThread(stgace-launcher07.diy.corp.ne1.yahoo.com:2181)]
INFO org.apache.zookeeper.ClientCnxn - Session establishment complete
on server stgace-launcher07.diy.corp.ne1.yahoo.com/98.138.236.98:2181,
sessionid = 0x1499166d6c30025, negotiated timeout = 40000
22:22:39.010 [main-EventThread] INFO
o.a.c.f.state.ConnectionStateManager - State change: CONNECTED
22:22:39.279 [main-SendThread(stgace-launcher07.diy.corp.ne1.yahoo.com:2181)]
DEBUG org.apache.zookeeper.ClientCnxn - Reading reply
sessionid:0x1499166d6c30025, packet:: clientPath:null serverPath:null
finished:false header:: 1,3 replyHeader:: 1,747,0 request::
'/drill,F response::
s{2,2,1415483705440,1415483705440,0,13,0,0,0,13,705}
22:22:39.280 [main] DEBUG o.a.d.e.c.zk.ZKClusterCoordinator - Starting
ZKClusterCoordination.
22:22:39.285 [main-SendThread(stgace-launcher07.diy.corp.ne1.yahoo.com:2181)]
DEBUG org.apache.zookeeper.ClientCnxn - Reading reply
sessionid:0x1499166d6c30025, packet:: clientPath:null serverPath:null
finished:false header:: 2,3 replyHeader:: 2,747,0 request::
'/drill,F response::
s{2,2,1415483705440,1415483705440,0,13,0,0,0,13,705}
22:22:39.287 [main-SendThread(stgace-launcher07.diy.corp.ne1.yahoo.com:2181)]
DEBUG org.apache.zookeeper.ClientCnxn - Reading reply
sessionid:0x1499166d6c30025, packet:: clientPath:null serverPath:null
finished:false header:: 3,3 replyHeader:: 3,747,0 request::
'/drill/drill-lih,F response::
s{3,3,1415483708586,1415483708586,0,15,0,0,0,1,671}
22:22:39.299 [main-SendThread(stgace-launcher07.diy.corp.ne1.yahoo.com:2181)]
DEBUG org.apache.zookeeper.ClientCnxn - Reading reply
sessionid:0x1499166d6c30025, packet:: clientPath:null serverPath:null
finished:false header:: 4,12 replyHeader:: 4,747,0 request::
'/drill/drill-lih,F response::
v{'4ee938e0-2d1a-413a-a222-017265b222c0},s{3,3,1415483708586,1415483708586,0,15,0,0,0,1,671}
22:22:39.309 [main-SendThread(stgace-launcher07.diy.corp.ne1.yahoo.com:2181)]
DEBUG org.apache.zookeeper.ClientCnxn - Reading reply
sessionid:0x1499166d6c30025, packet:: clientPath:null serverPath:null
finished:false header:: 5,4 replyHeader:: 5,747,0 request::
'/drill/drill-lih/4ee938e0-2d1a-413a-a222-017265b222c0,F response::
#a2434656539333865302d326431612d343133612d613232322d30313732363562323232633010ffffffb3ffffff88ffffff82ffffffb9ffffffa0291a36a287374676163652d6c61756e6368657230362e6469792e636f72702e6e65312e7961686f6f2e636f6d10ffffffa2fffffff2118ffffffa3fffffff2120ffffffa4fffffff21,s{653,653,1417458779337,1417458779337,0,0,0,92765138213601304,101,0,653}
22:22:39.336 [main-SendThread(stgace-launcher07.diy.corp.ne1.yahoo.com:2181)]
DEBUG org.apache.zookeeper.ClientCnxn - Reading reply
sessionid:0x1499166d6c30025, packet:: clientPath:/drill/drill-lih
serverPath:/drill/drill-lih finished:false header:: 6,12
replyHeader:: 6,747,0 request:: '/drill/drill-lih,T response::
v{'4ee938e0-2d1a-413a-a222-017265b222c0},s{3,3,1415483708586,1415483708586,0,15,0,0,0,1,671}
22:22:39.352 [main-SendThread(stgace-launcher07.diy.corp.ne1.yahoo.com:2181)]
DEBUG org.apache.zookeeper.ClientCnxn - Reading reply
sessionid:0x1499166d6c30025, packet::
clientPath:/drill/drill-lih/4ee938e0-2d1a-413a-a222-017265b222c0
serverPath:/drill/drill-lih/4ee938e0-2d1a-413a-a222-017265b222c0
finished:false header:: 7,4 replyHeader:: 7,747,0 request::
'/drill/drill-lih/4ee938e0-2d1a-413a-a222-017265b222c0,T response::
#a2434656539333865302d326431612d343133612d613232322d30313732363562323232633010ffffffb3ffffff88ffffff82ffffffb9ffffffa0291a36a287374676163652d6c61756e6368657230362e6469792e636f72702e6e65312e7961686f6f2e636f6d10ffffffa2fffffff2118ffffffa3fffffff2120ffffffa4fffffff21,s{653,653,1417458779337,1417458779337,0,0,0,92765138213601304,101,0,653}
22:22:39.380 [main-SendThread(stgace-launcher07.diy.corp.ne1.yahoo.com:2181)]
DEBUG org.apache.zookeeper.ClientCnxn - Reading reply
sessionid:0x1499166d6c30025, packet:: clientPath:null serverPath:null
finished:false header:: 8,12 replyHeader:: 8,747,0 request::
'/drill/drill-lih,F response::
v{'4ee938e0-2d1a-413a-a222-017265b222c0},s{3,3,1415483708586,1415483708586,0,15,0,0,0,1,671}
22:22:39.383 [main-SendThread(stgace-launcher07.diy.corp.ne1.yahoo.com:2181)]
DEBUG org.apache.zookeeper.ClientCnxn - Reading reply
sessionid:0x1499166d6c30025, packet:: clientPath:null serverPath:null
finished:false header:: 9,4 replyHeader:: 9,747,0 request::
'/drill/drill-lih/4ee938e0-2d1a-413a-a222-017265b222c0,F response::
#a2434656539333865302d326431612d343133612d613232322d30313732363562323232633010ffffffb3ffffff88ffffff82ffffffb9ffffffa0291a36a287374676163652d6c61756e6368657230362e6469792e636f72702e6e65312e7961686f6f2e636f6d10ffffffa2fffffff2118ffffffa3fffffff2120ffffffa4fffffff21,s{653,653,1417458779337,1417458779337,0,0,0,92765138213601304,101,0,653}
22:22:39.387 [main] DEBUG o.a.d.e.c.zk.ZKClusterCoordinator - Active
drillbit set changed. Now includes 1 total bits. New active
drillbits:
stgace-launcher06.diy.corp.ne1.yahoo.com:31010:31011:31012
22:22:39.418 [main] DEBUG i.n.c.MultithreadEventLoopGroup -
-Dio.netty.eventLoopThreads: 8
22:22:39.449 [main] DEBUG io.netty.channel.nio.NioEventLoop -
-Dio.netty.noKeySetOptimization: false
22:22:39.450 [main] DEBUG io.netty.channel.nio.NioEventLoop -
-Dio.netty.selectorAutoRebuildThreshold: 512
22:22:39.534 [main] DEBUG o.a.drill.exec.client.DrillClient -
Connecting to server stgace-launcher06.diy.corp.ne1.yahoo.com:31010
22:22:39.568 [main] DEBUG i.n.util.internal.ThreadLocalRandom -
-Dio.netty.initialSeedUniquifier: 0x6b4a02a017a39a90 (took 1 ms)
22:22:39.606 [main] DEBUG io.netty.buffer.ByteBufUtil -
-Dio.netty.allocator.type: unpooled
22:22:39.606 [main] DEBUG io.netty.buffer.ByteBufUtil -
-Dio.netty.threadLocalDirectBufferSize: 65536
22:22:39.636 [Client-1] DEBUG
i.n.u.i.JavassistTypeParameterMatcherGenerator - Generated:
io.netty.util.internal.__matchers__.io.netty.buffer.ByteBufMatcher
22:22:39.646 [Client-1] DEBUG
i.n.u.i.JavassistTypeParameterMatcherGenerator - Generated:
io.netty.util.internal.__matchers__.org.apache.drill.exec.rpc.OutboundRpcMessageMatcher
22:22:39.652 [Client-1] DEBUG
i.n.u.i.JavassistTypeParameterMatcherGenerator - Generated:
io.netty.util.internal.__matchers__.org.apache.drill.exec.rpc.InboundRpcMessageMatcher
22:22:39.677 [Client-1] DEBUG io.netty.util.Recycler -
-Dio.netty.recycler.maxCapacity.default: 262144
Connected to: Optiq (version unknown version)
Driver: Drill JDBC Driver (version unknown version)
Autocommit status: true
Transaction isolation: TRANSACTION_REPEATABLE_READ
sqlline version 1.0.2 by Marc Prud'hommeaux
0: jdbc:drill:zk=stgace-launcher07.diy.corp.n> select * from dfs.tmp.`test.csv`;
22:22:47.732 [Client-1] DEBUG o.a.d.e.rpc.user.QueryResultHandler -
Received QueryId part1: 3135002630360243693
part2: -6434486360344595063
succesfully. Adding listener
org.apache.drill.jdbc.DrillResultSet$Listener@25ae3fd
22:22:47.833 [Client-1] DEBUG org.apache.drill.jdbc.DrillResultSet -
Result arrived QueryResultBatch [header=query_id {
part1: 3135002630360243693
part2: -6434486360344595063
}
is_last_chunk: false
row_count: 0
def {
field {
major_type {
minor_type: VARCHAR
mode: REPEATED
}
name_part {
type: NAME
name: "columns"
}
}
}
, data=null]
22:22:47.844 [Client-1] DEBUG org.apache.drill.jdbc.DrillResultSet -
Result arrived QueryResultBatch [header=query_id {
part1: 3135002630360243693
part2: -6434486360344595063
}
is_last_chunk: false
row_count: 9
def {
record_count: 9
field {
major_type {
minor_type: VARCHAR
mode: REPEATED
}
name_part {
type: NAME
name: "columns"
}
value_count: 27
var_byte_length: 124
group_count: 9
buffer_length: 276
}
carries_two_byte_selection_vector: false
}
, data=DrillBuf(ridx: 0, widx: 276, cap: 276/276, unwrapped:
DrillBuf(ridx: 349, widx: 349, cap: 349/349, unwrapped:
UnsafeDirectLittleEndian(PooledUnsafeDirectByteBuf(ridx: 0, widx: 0,
cap: 349/349))))]
22:22:47.849 [Client-1] DEBUG org.apache.drill.jdbc.DrillResultSet -
Result arrived QueryResultBatch [header=query_id {
part1: 3135002630360243693
part2: -6434486360344595063
}
is_last_chunk: true
row_count: 0
def {
}
, data=null]
org/apache/hadoop/io/Text
0: jdbc:drill:zk=stgace-launcher07.diy.corp.n> 22:22:52.730
[main-SendThread(stgace-launcher07.diy.corp.ne1.yahoo.com:2181)] DEBUG
org.apache.zookeeper.ClientCnxn - Got ping response for sessionid:
0x1499166d6c30025
On Mon, Dec 1, 2014 at 7:27 PM, Jacques Nadeau <[email protected]> wrote:
> There isn't anything prebuilt. That being said, we use off the shelf
> sqlline. As such, you should be able to simply download sqlline and then
> use Drill's JDBC client jar file and run a remote sqlline.
>
> On Mon, Dec 1, 2014 at 12:12 PM, Jim Scott <[email protected]> wrote:
>
>> The sqlline utility takes a parameter for a zookeeper quorum to connect
>> to.
>>
>> When you run sqlline -u jdbc:drill:zk=local sqlline starts up an internal
>> local drillbit, but if you specify a remote zk then it will connect to a
>> remote drillbit.
>>
>> So, to answer your question, I believe you would install Drill on each of
>> those other 5 servers, but don't give them a configuration which allows
>> that installation to participate in the cluster.
>>
>> Alternatively, if you are asking if you can default the sqlline on those
>> servers with -u jdbc:drill:zk=server1,server2,server3, it doesn't appear
>> that the script currently supports that, but could likely be very simply
>> achieved with a tiny wrapper script.
>>
>>
>> On Mon, Dec 1, 2014 at 1:49 PM, David Tucker <[email protected]> wrote:
>>
>> >
>> > Is there a process for installing the sqlline tool independent of the
>> > complete drillbit ? The scenario I’m considering is a cluster of 10
>> nodes
>> > where only 5 would have drill-bits installed but users will expect to
>> > connect to the drill cluster arbitrarily from any node out of the 10.
>> >
>> > — David
>> >
>> >
>>
>>
>> --
>> *Jim Scott*
>> Director, Enterprise Strategy & Architecture
>> +1 (347) 746-9281
>>
>> <http://www.mapr.com/>
>> [image: MapR Technologies] <http://www.mapr.com>
>>
