Re: How to load data in Drill

Madhu Borkar Thu, 21 Nov 2013 18:13:10 -0800

Hi Ben,
I don't have the same sqlline script that you have mentioned. This I found
under bin
bin/sqlline




#!/bin/bash

ARGS=();
while [ $# -gt 0 ]
do
  case "$1" in
  -q) shift;
      QUERY=$1;;
  -e) shift;
      QUERY=$1;;
  -f) shift;
      FILE=$1;;
   *) ARGS+=($1);;
  esac
  shift
done

bin=`dirname "${BASH_SOURCE-$0}"`
bin=`cd "$bin">/dev/null; pwd`

. "$bin"/drill-config.sh

if [ -z $JAVA_HOME ]
then
  JAVA=`which java`
else
  JAVA=`find -L $JAVA_HOME -name java | head -n 1`
fi

if [ -e $JAVA ]; then
  echo ""
else
  echo "Java not found."
  exit 1
fi

$JAVA -version 2>&1 | grep "version" | egrep -e "1.7" > /dev/null
if [ $? -ne 0 ]; then
  echo "Java 1.7 is required to run Apache Drill."
  exit 1
fi

# get log directory
if [ "$DRILL_LOG_DIR" = "" ]; then
  export DRILL_LOG_DIR=/var/log/drill
fi


CP=$DRILL_HOME/jars/*:$CP
CP=$DRILL_HOME/lib/*:$CP
CP=$DRILL_HOME/contrib/*:$CP
CP=$HADOOP_CLASSPATH:$CP

CP=$DRILL_CONF_DIR:$CP

DRILL_SHELL_JAVA_OPTS="$DRILL_SHELL_JAVA_OPTS
-Dlog.path=$DRILL_LOG_DIR/sqlline.log"

if [ -n "$QUERY" ]
then
  echo $QUERY | exec $JAVA $DRILL_SHELL_JAVA_OPTS $DRILL_JAVA_OPTS -cp $CP
sqlline.SqlLine "${ARGS[@]}"
elif [ -n "$FILE" ]
then
  exec $JAVA $DRILL_SHELL_JAVA_OPTS $DRILL_JAVA_OPTS -cp $CP
sqlline.SqlLine "${ARGS[@]}" --run=$FILE
else
  exec $JAVA $DRILL_SHELL_JAVA_OPTS $DRILL_JAVA_OPTS -cp $CP
sqlline.SqlLine "${ARGS[@]}"
fi
-----------------------
Is this because I installed from binary?
Thanks
-Madhu

On Thu, Nov 21, 2013 at 9:50 AM, Ben Becker <[email protected]> wrote:

> Hi Madhu,
>
> Could you attach the complete logs during this run?  The easiest way is
> probably to edit the sqlline shell script and remove the '
> -Dlogback.configurationFile=...' argument like so:
>
> -exec java -Xmx2G
> -Dlogback.configurationFile=./sqlparser/src/test/resources/logback.xml -cp
> ./:"$(cat .classpath)" sqlline.SqlLine --verbose=true "$@"
> *+exec java -Xmx2G -cp ./:"$(cat .classpath)" sqlline.SqlLine
> --verbose=true "$@"*
>
> This will cause all log output to be directed to the console.
>  Alternatively, you may want to log to a file or another facility by
> editing the aforementioned logback.xml file.  One last note -- although
> unlikely, parts of the dataset may be included in the logs, so be sure to
> redact any sensitive information from the logs before posting.
>
> Thanks,
> Ben
>
>
>
> On Tue, Nov 19, 2013 at 6:36 PM, Madhu Borkar <[email protected]>
> wrote:
>
> > Ben,
> > Here is log snippet using lilith..
> > <?xml version='1.0' encoding='UTF-8'?>
> > <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "
> > http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd";>
> > <html xmlns='http://www.w3.org/1999/xhtml
> '><head><title>Title</title><link
> > href='detailsView.css' rel='stylesheet'
> > type='text/css'/></head><body><table><tr
> > class='even'><th>Message</th><td><pre>Exception during
> > query</pre></td></tr><tr class='odd'><th>Throwable</th><td
> > class='throwableContainer'><div class='throwable'><a
> >
> >
> href='ste://org.apache.drill.exec.rpc.user.QueryResultHandler.batchArrived(QueryResultHandler.java:72)'>org.apache.drill.exec.rpc.RpcException</a><pre><a
> >
> >
> href='ste://org.apache.drill.exec.rpc.user.QueryResultHandler.batchArrived(QueryResultHandler.java:72)'>Remote
> > failure while running query.[error_id:
> > "b9ce54dd-085c-435d-b12f-e9896c60e864"
> > endpoint {
> >   address: "Node-0.etouch.net"
> >   user_port: 31010
> >   bit_port: 32011
> > }
> > error_type: 0
> > message: "Failure while running fragment. &lt; NullPointerException"
> > ]</a></pre><hr/><a
> >
> >
> href='ste://org.apache.drill.exec.rpc.user.QueryResultHandler.batchArrived(QueryResultHandler.java:72)'>at&nbsp;org.apache.drill.exec.rpc.user.QueryResultHandler.batchArrived(QueryResultHandler.java:72)&nbsp;~[java-exec-1.0.0-m1-rebuffed.jar:1.0.0-m1]</a><a
> >
> >
> href='ste://org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:79)'>at&nbsp;org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:79)&nbsp;~[java-exec-1.0.0-m1-rebuffed.jar:1.0.0-m1]</a><a
> >
> >
> href='ste://org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:48)'>at&nbsp;org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:48)&nbsp;~[java-exec-1.0.0-m1-rebuffed.jar:1.0.0-m1]</a><a
> >
> >
> href='ste://org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:33)'>at&nbsp;org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:33)&nbsp;~[java-exec-1.0.0-m1-rebuffed.jar:1.0.0-m1]</a><a
> >
> >
> href='ste://org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:142)'>at&nbsp;org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:142)&nbsp;~[java-exec-1.0.0-m1-rebuffed.jar:1.0.0-m1]</a><a
> >
> >
> href='ste://org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:127)'>at&nbsp;org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:127)&nbsp;~[java-exec-1.0.0-m1-rebuffed.jar:1.0.0-m1]</a><a
> >
> >
> href='ste://io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)'>at&nbsp;io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)&nbsp;~[netty-codec-4.0.7.Final.jar:na]</a><a
> >
> >
> href='ste://io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:334)'>at&nbsp;io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:334)&nbsp;~[netty-transport-4.0.7.Final.jar:na]</a><a
> >
> >
> href='ste://io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:320)'>at&nbsp;io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:320)&nbsp;~[netty-transport-4.0.7.Final.jar:na]</a><a
> >
> >
> href='ste://io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)'>at&nbsp;io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)&nbsp;~[netty-codec-4.0.7.Final.jar:na]</a><a
> >
> >
> href='ste://io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:334)'>at&nbsp;io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:334)&nbsp;~[netty-transport-4.0.7.Final.jar:na]</a><a
> >
> >
> href='ste://io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:320)'>at&nbsp;io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:320)&nbsp;~[netty-transport-4.0.7.Final.jar:na]</a><a
> >
> >
> href='ste://io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:173)'>at&nbsp;io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:173)&nbsp;~[netty-codec-4.0.7.Final.jar:na]</a><a
> >
> >
> href='ste://io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:334)'>at&nbsp;io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:334)&nbsp;~[netty-transport-4.0.7.Final.jar:na]</a><a
> >
> >
> href='ste://io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:320)'>at&nbsp;io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:320)&nbsp;~[netty-transport-4.0.7.Final.jar:na]</a><a
> >
> >
> href='ste://io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:785)'>at&nbsp;io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:785)&nbsp;~[netty-transport-4.0.7.Final.jar:na]</a><a
> >
> >
> href='ste://io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:100)'>at&nbsp;io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:100)&nbsp;~[netty-transport-4.0.7.Final.jar:na]</a><a
> >
> >
> href='ste://io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:497)'>at&nbsp;io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:497)&nbsp;~[netty-transport-4.0.7.Final.jar:na]</a><a
> >
> >
> href='ste://io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:465)'>at&nbsp;io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:465)&nbsp;~[netty-transport-4.0.7.Final.jar:na]</a><a
> >
> >
> href='ste://io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:359)'>at&nbsp;io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:359)&nbsp;~[netty-transport-4.0.7.Final.jar:na]</a><a
> >
> >
> href='ste://io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)'>at&nbsp;io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)&nbsp;~[netty-common-4.0.7.Final.jar:na]</a><a
> >
> >
> href='ste://java.lang.Thread.run(Thread.java:744)'>at&nbsp;java.lang.Thread.run(Thread.java:744)&nbsp;~[na:1.7.0_45]</a></div></td></tr><tr
> > class='even'><th>Level</th><td>ERROR</td></tr><tr
> >
> >
> class='odd'><th>Logger</th><td>org.apache.drill.sql.client.full.ResultEnumerator</td></tr><tr
> > class='even'><th>Thread</th><td>main (id=1)</td></tr><tr
> > class='odd'><th>Timestamp</th><td>2013-11-19
> 18:26:42.023-0800</td></tr><tr
> > class='even'><th>Call&nbsp;Location</th><td><a
> >
> >
> href='ste://org.apache.drill.sql.client.full.ResultEnumerator.moveNext(ResultEnumerator.java:43)'>at&nbsp;org.apache.drill.sql.client.full.ResultEnumerator.moveNext(ResultEnumerator.java:43)</a></td></tr><tr
> >
> >
> class='odd'><th>Logger&nbsp;Context</th><td><table><tr><th>Name</th><td>default</td></tr><tr><th>Birthtime</th><td>2013-11-19
> >
> >
> 18:26:06.371-0800</td></tr><tr><th>Properties</th><td><table><tr><th>Key</th><th>Value</th></tr><tr><td>HOSTNAME</td><td>
> > Node-0.etouch.net</td></tr></table></td></tr></table></td></tr><tr
> >
> >
> class='even'><th>Source&nbsp;ID</th><td>127.0.0.1-20131119T182606709</td></tr><tr
> > class='odd'><th>Local&nbsp;ID</th><td>71</td></tr></table></body></html>
> >
> >
> > On Tue, Nov 19, 2013 at 9:34 AM, Ben Becker <[email protected]>
> wrote:
> >
> > > Hi Madhu,
> > >
> > > Could you report which version you're using (e.g. 1.0.0-m1.rc3), and
> > attach
> > > the logs when this happens?  You can use Lilith to view the logs if you
> > > like (see instructions at the bottom of this
> > > page<
> > >
> >
> https://cwiki.apache.org/confluence/display/DRILL/How+To+Run+Drill+with+Sample+Data#
> > > >
> > > ).
> > >
> > > Thanks,
> > > Ben
> > >
> > >
> > > On Mon, Nov 18, 2013 at 10:16 PM, Madhu Borkar <[email protected]
> > > >wrote:
> > >
> > > > I have devlopment MapR cluster. I have installed Apache Drill using
> > > > binaries. I could run queries with sample-data.
> > > >
> > > > I decided to load test data using pig. Here is my code
> > > >
> > > > ---------------------------------
> > > >
> > > > REGISTER file:/home/mapr/lib/parquet-pig-bundle-1.2.5.jar
> > > >
> > > > data = LOAD '$DATA'
> > > >
> > > >                 USING PigStorage(',')
> > > >
> > > > as (
> > > >
> > > >       user:    chararray,
> > > >
> > > >       street: chararray,
> > > >
> > > >       state:          chararray,
> > > >
> > > >       city:           chararray,
> > > >
> > > >       type:  chararray
> > > >
> > > >   );
> > > >
> > > >
> > > >
> > > > A = LIMIT data 3;
> > > >
> > > > dump A;
> > > >
> > > > fs -rmr /mapr-hive/parquet/data;
> > > >
> > > > STORE data INTO '/mapr-hive/parquet/data' using
> > > > parquet.pig.ParquetStorer();
> > > >
> > > > ----------------------
> > > >
> > > > Pig script writes to output. I moved data under apache drill
> directory.
> > > >
> > > >
> > > >
> > > > select * from "real-data/users.parquet";
> > > >
> > > >
> > > >
> > > > but when I try to query this data I get rpc error?
> > > >
> > > > Query failed: org.apache.drill.exec.rpc.RpcException: Remote failure
> > > while
> > > > running query.[error_id: "04ac52e4-b49e-45b8-82c5-cf480154eefd"
> > > >
> > > > endpoint {
> > > >
> > > >   address: "Node.etouch.net"
> > > >
> > > >   user_port: 31010
> > > >
> > > >   bit_port: 32011
> > > >
> > > > }
> > > >
> > > > error_type: 0
> > > >
> > > > message: "Failure while running fragment. < NullPointerException"
> > > >
> > > > ]
> > > >
> > > >
> > > >
> > > > Please, provide me help how to load data in Apache Drill.?
> > > >
> > > > Is there MapR configuration needed for Drill?
> > > >
> > > > Regards
> > > >
> > >
> >
>

Re: How to load data in Drill

Reply via email to