I should have added that there is nothing wrong with the dual plugin approach on MapR. Works fine and it is up to you as a matter of personal choice which is better.
On Fri, May 22, 2015 at 4:46 PM, Ted Dunning <[email protected]> wrote: > > As a special case, with MapR, you can access all clusters in an > administrative group by making sure that you have /mapr/<cluster-name> at > the beginning of your path names. THis means that you can simply use > different workspaces, or a workspace with a path consisting only of /mapr > and still access files and tables on multiple clusters. > > The dual plugin approach would be required for HDFS based clusters. > > > > On Fri, May 22, 2015 at 9:49 AM, Abhishek Girish <[email protected]> wrote: > >> I tried out Tomar's steps on MapR and it was pretty straight-forward. >> >> I have drill installed on one cluster. The only change I made was to add a >> new storage plug-in "dfs2" (duplicating the default dfs). I edited the >> connection string and changed "maprfs:///" to "maprfs://<IP>". And when i >> connected to drill via sqlline (no changes here), I was able to access the >> remote file system by simply using the the full path of the file, prefixed >> with "dfs2.". >> >> In case of HDFS, I'm assuming the steps required must be similar, except >> the connection string (hdfs://<IP>:<port>) >> >> Also, as Andrews mentioned, there looks to be a typo in your query - that >> could very well be the issue. >> >> Thanks, >> Abhishek >> >> On Fri, May 22, 2015 at 8:27 AM, Andries Engelbrecht < >> [email protected]> wrote: >> >> > In step2 you need to have a back tick ` at the end of par and not a >> single >> > quote ‘ . >> > >> > It has been mentioned that it may not work to just point to the name >> node >> > on a remote cluster. I have not tried it, but suspect their may be >> various >> > issues with the HDFS plug in and how you are trying to use it. >> > >> > Perhaps if you can explain why you are trying to do this, there may be >> > other alternatives to explore. >> > >> > What Hadoop distro are you using? >> > >> > —Andries >> > >> > >> > On May 22, 2015, at 8:17 AM, Alan Miller <[email protected]> >> wrote: >> > >> > > Thanks Shiran, >> > > >> > > I tried that but get the same error (see below). >> > > >> > > Also, strangely I couldn't create the hdfs plugin in one step by using >> > the same >> > > config as the "dfs" plugin and changing the connection string. The UI >> > says Invalid JSON... >> > > I had to create the hdfs plugin in 2 steps. First using the same >> config >> > as the dfs plugin. >> > > Then updated the hdfs config, by changing the connection string >> > > >> > > After adding the hdfs plugin with the same config as dfs (but >> different >> > connection >> > > string ("connection": "hdfs://10.10.10.10:9000/",) I tried this >> > > >> > > 1. Copied the file from node1 to remote HDFS >> > > [alan@node1 drill]$ hdfs dfs -fs hdfs://10.10.10.10:9000/ >> > -copyFromLocal ~/test.par /tmp >> > > [alan@node1 drill]$ hdfs dfs -fs hdfs://10.10.10.10:9000/ -ls >> > /tmp/test.par >> > > -rw-r--r-- 1 alan supergroup 4947359 2015-05-22 08:09 >> > /tmp/test.par >> > > >> > > 2. From drill on node1 >> > > [alan@node1 drill]$ /opt/drill/bin/drill-localhost >> > > apache drill 1.0.0 >> > > "json ain't no thang" >> > > 0: jdbc:drill:drillbit=localhost> use hdfs; >> > > +-------+-----------------------------------+ >> > > | ok | summary | >> > > +-------+-----------------------------------+ >> > > | true | Default schema changed to [hdfs] | >> > > +-------+-----------------------------------+ >> > > 1 row selected (0.422 seconds) >> > > 0: jdbc:drill:drillbit=localhost> select * from >> > hdfs.root.`/tmp/test.par' limit 5; >> > > Error: PARSE ERROR: Lexical error at line 1, column 55. >> Encountered: >> > <EOF> after : "`/tmp/test.par\' limit 5" >> > > >> > > [Error Id: 1f793d84-62be-4145-bfcf-2ec3da9cb021 on >> > node1.mycompany.com:31010] (state=,code=0) >> > > 0: jdbc:drill:drillbit=localhost> >> > > 0: jdbc:drill:drillbit=localhost> !quit >> > > Closing: >> > org.apache.drill.jdbc.DrillJdbc41Factory$DrillJdbc41Connection >> > > >> > > Alan >> > >> > >> >> >> -- >> >> Abhishek Girish >> >> Senior Software Engineer >> >> (408) 476-9209 >> >> <http://www.mapr.com/> >> > >
