This information is taken from https://issues.apache.org/jira/browse/DRILL-1075
Here’s a sample configuration for HDFS:
{
"type" : "file",
"enabled" : true,
"connection" : "hdfs://10.10.30.156:8020/",
“workspaces" : {
“root : {
"location" : "/user/root/drill",
"writable" : true,
"storageformat" : "null"
}
},
“formats" : {
“json” : {
"type" : “json"
}
}
}
Make sure these packages are in the class path (changes based on installation):
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop/hadoop-annotations-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop/hadoop-auth-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop/hadoop-common-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop-mapreduce/hadoop-mapreduce-client-core-2.0.0-cdh4.7.0.jar
And point to the correct Zookeeper in drill-override.conf
drill.exec : { cluster-id: "working_cdh_drill" zk.connect: "10.10.30.156:2181” }
There is an open issue to add the packages mentioned above to the class path
automatically:
https://issues.apache.org/jira/browse/DRILL-1160
Thank you,
Sudheesh Katkam
On Jul 31, 2014, at 7:47 AM, Bert M. <[email protected]> wrote:
>
> Neeraja Rentachintala <nrentachintala@...> writes:
>
>>
>> Are you able to query the local file system successfully.?
>> The doc
>>
> https://cwiki.apache.org/confluence/display/DRILL/Connecting+to+Data+Sources
> #ConnectingtoD
> ataSources-QueryingaFileSystem
>> has info on the sample hdfs connection under 'querying file system'.
>> Is this not working for you.
>>
>> On Wed, Jul 30, 2014 at 11:49 AM, Bert M <bgmlist@...> wrote:
>>
>>> Hello - I get a "Unable to create/ update plugin:" error on any attempts
>>> to add an HDFS storage destination via the Storage-plugin configuration UI.
>>> An example config is:
>>> { "type": "file", "enabled": true, "connection":
>>> "hdfs://ip-xxx-yyy-zzz.ec2.internal:8020",}
>>>
>>> I've also tried a plain "connection": "hdfs:///" variation too, without
>>> much success.This is with both, the binary distribution for drill-M2 as
>>> well as a compiled version from the top of the tree, deployed on a CDH5.1
>>> stack.
>>> I couldn't find any docs or examples beyond the simple one in the wiki
>>> which shows the second plain hdfs:/// variation. Could someone help point
>>> me in the right direction?
>>> Thanks,B
>>
>
> Yes, I'm able to create or modify storage configs with local filesystems
> and use them successfully. The problem is that I cannot create/update
> a storage config to use an hdfs location (and yes, that
> hdfs location does work correctly, both directly and in other
> hadoop based applications).
>
>
> This behavior is exhibited with the example config from the doc too, viz.:
> {
> "type": "file",
> "enabled": true,
> "connection": "hdfs:///",
> "workspaces": {
> "root": {
> "location": "/",
> "writable": false,
> "storageformat": null
> }
> }
> }
>
>
> This works fine with "connection": "file:///", but the following error is
> generated with the "connection": "hdfs:///"
>
> 2014-07-30 18:39:36,007 [qtp353753254-71] DEBUG
> o.a.d.e.server.rest.StorageResources - Unable to create/ update plugin: zfs
>
>
> Are any other settings required to enable HDFS access in Drill?
>
> Thanks,
> B
>
>
