Yes. That is the page I found it on. I am working with it tonight and am
able to get the dfs plugin working on EMR using the setup you described.
Thanks again for the help. We will be deploying Drill on both our always on
MapR cluster as well as on-demand AWS EMR clusters, so have both use cases.

- Dave

On Thu, Oct 13, 2016 at 9:25 PM, Ted Dunning <ted.dunn...@gmail.com> wrote:

> David,
>
> It isn't misleading on a MapR cluster. To limit the number of problems with
> programs that hard-code hdfs: at the beginning of file names, the default
> is to assign "hdfs" to the MapR FS driver. As such, on such a system you
> don't need a namenode address in the URL. My guess is that a MapR user
> wrote (and possibly even tested) wrote that sentence.
>
> A clarification is clearly needed.
>
> Btw... is this the page you found that info on?
>
> https://drill.apache.org/docs/file-system-storage-plugin/
>
>
>
>
> On Thu, Oct 13, 2016 at 7:16 AM, David Kincaid <kincaid.d...@gmail.com>
> wrote:
>
> > I understood what Ted said and saw that in the documentation. I was
> > pointing out that right after that it says on the local cluster you
> simply
> > need to change file:/// to hdfs://. That is a bit misleading.
> >
> > On Thu, Oct 13, 2016 at 9:12 AM, Andries Engelbrecht <
> > aengelbre...@maprtech.com> wrote:
> >
> > > See
> > >
> > > http://drill.apache.org/docs/file-system-storage-plugin/ <
> > > http://drill.apache.org/docs/file-system-storage-plugin/>
> > >
> > > "connection": "hdfs://<IP Address>:<Port>/"
> > >
> > >
> > > As Ted stated you need to point to the name node for HDFS.
> > >
> > >
> > > --Andries
> > >
> > >
> > > > On Oct 13, 2016, at 5:04 AM, David Kincaid <kincaid.d...@gmail.com>
> > > wrote:
> > > >
> > > > Thanks, Ted. The full URL I was using was http://. I'll give your
> > > > suggestion a try when I'm able to work on this again tonight. I
> guess I
> > > > took the documentation too literally when it said "To query a file on
> > > HDFS
> > > > from a node on the cluster, you can simply change the connection from
> > > > file:/// to hdfs:// in the dfs storage plugin."
> > > > Thanks again,
> > > > Dave
> > > >
> > > >
> > > > On Thu, Oct 13, 2016 at 12:39 AM, Ted Dunning <ted.dunn...@gmail.com
> >
> > > wrote:
> > > >
> > > >> What is the full URL you used?
> > > >>
> > > >> With hdfs://, you need to supply a name node address.
> > > >>
> > > >> With file://, you don't.
> > > >>
> > > >> Contrarily, with maprfs:// you don't need an address since it is
> > > implied in
> > > >> the client connection.
> > > >>
> > > >>
> > > >>
> > > >> On Wed, Oct 12, 2016 at 6:29 PM, David Kincaid <
> > kincaid.d...@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> I have an Amazon EMR cluster launched with Drill loaded. I'm trying
> > to
> > > >>> configure the dfs storage plugin to use HDFS. The docs say that I
> > > should
> > > >>> simply need to change the "connection" setting from "file:///" to
> > > >> "hdfs://"
> > > >>> in order to use HDFS on the cluster that Drill is running on.
> > However,
> > > >> when
> > > >>> I do this and try to run a query I get an error that says
> > > >>> "org.apache.drill.common.exceptions.UserRemoteException:
> > > >>> SYSTEM ERROR: URISyntaxException: Expected authority at index 7:
> > > hdfs://
> > > >>> [Error Id: f9e6c674-4dd7-4c5d-b9a8-95b64b9dbaa3"
> > > >>>
> > > >>> Am I doing something wrong or is there an issue here?
> > > >>>
> > > >>> Thanks,
> > > >>>
> > > >>> Dave
> > > >>>
> > > >>
> > >
> > >
> >
>

Reply via email to