Hi Arthur, Currently schema of FileSystem storage plugin doesn't allow you to any configuration property other than "fs.default.name" (basically the connection string) for configuration. It would be good if we have "config" section similar to hive or hbase storage plugins. Please log a jira for improvement. One workaround is copy core-site.xml/hdfs-site.xml for hadoop installation into a directory which is on the Drills classpath. This way whenever a Configuration object is instantiated it gets all properties including HDFS HA properties from core-site.xml/hdfs-site.xml.
Thanks Venki On Thu, Jul 16, 2015 at 1:43 AM, Arthur Chan <[email protected]> wrote: > On Wed, Jun 17, 2015 at 10:47 PM, Arthur Chan <[email protected]> > wrote: > > > Hi, > > > > Thanks! > > > > There is one thing about "connection" that I feel confused: > > > > In the HA Hadoop Cluster, instead of use <ip_address> and <port>, we use > > <zookeeper QuorumPeerMain> and use <dfs.nameservices> to manage the > > connection to HA HDFS, > > > > > > Below is the hdfs-site.xml here: > > > > ============== > > > > <property> > > > > <name>dfs.nameservices</name> > > > > <value>mycluster</value> > > > > <final>true</final> > > > > </property> > > ================ > > > > > > How to define ""connection" properly in my case? > > > > Regards > > > > > > > > > > On Wed, Jun 17, 2015 at 10:35 PM, Akif Khan <[email protected]> > > wrote: > > > >> This is the config for HDFS : > >> > >> { > >> "type": "file", > >> "enabled": true, > >> "connection": "hdfs://<ip_address>:8020/", > >> "workspaces": { > >> "root": { > >> "location": "/user/data", > >> "writable": true, > >> "defaultInputFormat": null > >> } > >> }, > >> "formats": { > >> "psv": { > >> "type": "text", > >> "extensions": [ > >> "tbl" > >> ], > >> "delimiter": "|" > >> }, > >> "csv": { > >> "type": "text", > >> "extensions": [ > >> "csv" > >> ], > >> "delimiter": "," > >> }, > >> "tsv": { > >> "type": "text", > >> "extensions": [ > >> "tsv" > >> ], > >> "delimiter": "\t" > >> }, > >> "parquet": { > >> "type": "parquet" > >> }, > >> "json": { > >> "type": "json" > >> }, > >> "avro": { > >> "type": "avro" > >> } > >> } > >> } > >> > >> On Wed, Jun 17, 2015 at 8:03 PM, Arthur Chan <[email protected]> > >> wrote: > >> > >> > Hi, > >> > > >> > I am new to Drill, could anyone advise how to configure HDFS storage > and > >> > how to do smoke tests about Drill on HDFS? > >> > > >> > ============= > >> > { > >> > "type": "file", > >> > "enabled": true, > >> > "connection": "file:///", > >> > "workspaces": { > >> > "root": { > >> > "location": "/", > >> > "writable": true, > >> > "defaultInputFormat": null > >> > }, > >> > "tmp": { > >> > "location": "/tmp", > >> > "writable": true, > >> > "defaultInputFormat": null > >> > } > >> > }, > >> > "formats": { > >> > "psv": { > >> > "type": "text", > >> > "extensions": [ > >> > "tbl" > >> > ], > >> > "delimiter": "|" > >> > }, > >> > "csv": { > >> > "type": "text", > >> > "extensions": [ > >> > "csv" > >> > ], > >> > "delimiter": "," > >> > }, > >> > "tsv": { > >> > "type": "text", > >> > "extensions": [ > >> > "tsv" > >> > ], > >> > "delimiter": "\t" > >> > }, > >> > "parquet": { > >> > "type": "parquet" > >> > }, > >> > "json": { > >> > "type": "json" > >> > }, > >> > "avro": { > >> > "type": "avro" > >> > } > >> > } > >> > } > >> > ============= > >> > > >> > >> > >> > >> -- > >> Regards > >> > >> *Akif Khan* > >> *InnovAccer Inc.* > >> *www.innovaccer.com <http://www.innovaccer.com>* > >> *+91 8802290360* > >> > > > > >
