Thanks Andries. I updated the config accordingly. On Mon, May 11, 2015 at 8:26 PM, Andries Engelbrecht < [email protected]> wrote:
> instead of file:/// use maprfs:/// (I assume it is a mapr cluster) > > If you do that you don’t need to use /mapr/clustername in front of the > location path, but you can rather just directly point to the maprfs > directory. > > See the sample I posted vs the storage plug in you had configured. > > The default DFS plugin should also show you the correct config info. > > —Andries > > On May 11, 2015, at 5:21 PM, Minnow Noir <[email protected]> wrote: > > > I'm not sure I understand what you mean, Andries. What field and value > are > > you suggesting that I change? If there's a better way to do this, I > would > > like to do it the right way, even if it's not the issue in question. > > > > Thanks > > > > On Mon, May 11, 2015 at 8:14 PM, Andries Engelbrecht < > > [email protected]> wrote: > > > >> You set up the Storage plugin to use the loopback NFS to read the file, > >> which is not the best way to do it. > >> > >> Why not use the MapR-FS or HDFS plug ins , this is a setup that uses the > >> MapR-FS plugin which will be considerably better than the loopback NFS > for > >> Drill > >> > >> > >>> { > >>> "type": "file", > >>> "enabled": true, > >>> "connection": “maprfs:///", > >>> "workspaces": { > >>> "schema1": { > >>> "location": "/user/theUser/test", > >>> "writable": true, > >>> "defaultInputFormat": null > >>> } > >>> }, > >> > >> > >> —Andries > >> > >> > >> On May 11, 2015, at 5:07 PM, Minnow Noir <[email protected]> wrote: > >> > >>> Hanifi: The web UI is extremely slow, and the style sheets aren't > >> loading, > >>> but here is what I see for "show databases" > >>> > >>> Major FragmentMinor Fragments ReportingFirst StartLast StartFirst > EndLast > >>> Endtmintavgtmaxlast updatelast progressmemmax 00-xx-xx1 / > >>> 10.213s0.213s0.670s0.670s0.457s0.457s0.457s12:15:3012:15:303MB > >>> Major Fragment: 00-xx-xx > >>> < > >> > http://174.129.247.50:8047/profiles/2aaf002d-0351-e983-b9a1-51da33885d87#fragment-0 > >>> > >>> Minor FragmentHostStartEndTotal TimeMax RecordsMax BatchesLast > >> UpdateLast > >>> ProgressPeak MemoryState > >> 00-00-xxip-10-101-197-197.ec2.internal0.213s0.670s > >>> 0.457s6112:15:3012:15:303MBFINISHED > >>> Operator Profiles > >>> Overview > >>> < > >> > http://174.129.247.50:8047/profiles/2aaf002d-0351-e983-b9a1-51da33885d87#operator-overview > >>> > >>> OperatorTypeSetup (min)Setup (avg)Setup (max)Process (min)Process > >>> (avg)Process > >>> (max)Wait (min)Wait (avg)Wait (max)Mem (avg)Mem (max) > >> 00-xx-00SCREEN0.000s > >>> 0.000s0.000s0.000s0.000s0.000s0.000s0.000s0.000s-- > 00-xx-01PROJECT0.422s > >>> 0.422s0.422s0.001s0.001s0.001s0.000s0.000s0.000s-- 00-xx-02 > >>> > >> > INFO_SCHEMA_SUB_SCAN0.000s0.000s0.000s0.020s0.020s0.020s0.000s0.000s0.000s > >>> 17MB17MB > >>> 00-xx-00 - SCREEN > >>> < > >> > http://174.129.247.50:8047/profiles/2aaf002d-0351-e983-b9a1-51da33885d87#operator-0-0 > >>> > >>> Minor FragmentSetupProcessWaitMax BatchesMax RecordsPeak Mem 00-00-00 > >>> 0.000s0.000s0.000s16- > >>> 00-xx-01 - PROJECT > >>> < > >> > http://174.129.247.50:8047/profiles/2aaf002d-0351-e983-b9a1-51da33885d87#operator-0-1 > >>> > >>> Minor FragmentSetupProcessWaitMax BatchesMax RecordsPeak Mem 00-00-01 > >>> 0.422s0.001s0.000s16- > >>> 00-xx-02 - INFO_SCHEMA_SUB_SCAN > >>> < > >> > http://174.129.247.50:8047/profiles/2aaf002d-0351-e983-b9a1-51da33885d87#operator-0-2 > >>> > >>> Minor FragmentSetupProcessWaitMax BatchesMax RecordsPeak Mem > >>> 00-00-020.000s0.020s0.000s0017MB > >>> > >>> Andries: That storage plugin is very simple. I've obfuscated some > >>> proprietary info. (Note that the web UI was slow/hanging before I > added > >>> the storage plugin.) > >>> > >>> { > >>> "type": "file", > >>> "enabled": true, > >>> "connection": "file:///", > >>> "workspaces": { > >>> "schema1": { > >>> "location": "/mapr/theClusterName/user/theUser/test", > >>> "writable": true, > >>> "defaultInputFormat": null > >>> } > >>> }, > >>> "formats": { > >>> "csv": { > >>> "type": "text", > >>> "extensions": [ > >>> "csv" > >>> ], > >>> "delimiter": "," > >>> }, > >>> "tsv": { > >>> "type": "text", > >>> "extensions": [ > >>> "tsv" > >>> ], > >>> "delimiter": "\t" > >>> }, > >>> "parquet": { > >>> "type": "parquet" > >>> } > >>> } > >>> } > >>> > >>> Unfortunately, because the Web UI is having issues, it is presenting an > >>> unstyled page with no disable links for the storage plugin so I can't > >>> disable it. With the web UI not working, I don't know how to delete the > >>> plugin. > >>> > >>> Abdel: There are no errors in the logs. > >>> > >>> > >>> On Mon, May 11, 2015 at 7:22 PM, Hanifi Gunes <[email protected]> > >> wrote: > >>> > >>>> I would be interested in knowing where the time has been spent. Can > you > >>>> inspect query profile from web ui for `show databases` and let us know > >> how > >>>> the profile looks like? > >>>> > >>>> On Mon, May 11, 2015 at 4:10 PM, Abdel Hakim Deneche < > >>>> [email protected]> > >>>> wrote: > >>>> > >>>>> any errors in the logs ? > >>>>> > >>>>> On Mon, May 11, 2015 at 4:03 PM, Minnow Noir <[email protected]> > >>>> wrote: > >>>>> > >>>>>> Yes. The bit and server are idle, with no jobs of any sort running > on > >>>>>> them. > >>>>>> > >>>>>> I just did started sqlline and it happened again. I had to cancel > the > >>>>> show > >>>>>> databases command after 30s. > >>>>>> > >>>>>> sqlline version 1.1.6 > >>>>>> 0: jdbc:drill:zk=theServer:5181> show databases; > >>>>>> +-------------+ > >>>>>> | SCHEMA_NAME | > >>>>>> +-------------+ > >>>>>> | INFORMATION_SCHEMA | > >>>>>> | cp.default | > >>>>>> | dfs.default | > >>>>>> | dfs.root | > >>>>>> | dfs.tmp | > >>>>>> | test.default | > >>>>>> | test.schema1 | > >>>>>> | sys | > >>>>>> +-------------+ > >>>>>> 8 rows selected (29.264 seconds) > >>>>>> > >>>>>> On Mon, May 11, 2015 at 6:57 PM, Hanifi Gunes <[email protected]> > >>>>> wrote: > >>>>>> > >>>>>>> Can you confirm if this happens even when the particular 0.9 bit is > >>>>> idle? > >>>>>>> > >>>>>>> -Hanifi > >>>>>>> > >>>>>>> On Mon, May 11, 2015 at 3:53 PM, Minnow Noir <[email protected] > > > >>>>>> wrote: > >>>>>>> > >>>>>>>> Experiencing some slowness after upgrading one of our servers from > >>>>> 0.8 > >>>>>> to > >>>>>>>> 0.9. In particular, despite restarting the Drillbit, and even the > >>>>>>> server, > >>>>>>>> several times, the Drill web UI and sqlline are both so slow as to > >>>> be > >>>>>>>> unusable. > >>>>>>>> > >>>>>>>> For example, doing a show databases hangs indefinitely, until I > >>>>> CTRL-C > >>>>>>> the > >>>>>>>> command (e.g., after a minute). > >>>>>>>> > >>>>>>>> show databases; > >>>>>>>> +-------------+ > >>>>>>>> | SCHEMA_NAME | > >>>>>>>> +-------------+ > >>>>>>>> | INFORMATION_SCHEMA | > >>>>>>>> | cp.default | > >>>>>>>> | dfs.default | > >>>>>>>> | dfs.root | > >>>>>>>> | dfs.tmp | > >>>>>>>> | test.default | > >>>>>>>> | test.schema1 | > >>>>>>>> | sys | > >>>>>>>> +-------------+ > >>>>>>>> 8 rows selected (59.044 seconds) > >>>>>>>> > >>>>>>>> I tried changing schemas, but had to cancel after waiting several > >>>>>>> minutes. > >>>>>>>> > >>>>>>>> use test.schema1; > >>>>>>>> +------------+------------+ > >>>>>>>> | ok | summary | > >>>>>>>> +------------+------------+ > >>>>>>>> | true | Default schema changed to 'test.schema1' | > >>>>>>>> +------------+------------+ > >>>>>>>> 1 row selected (208.089 seconds) > >>>>>>>> > >>>>>>>> There are no errors or obvious issues in the logs. The server has > >>>>>> plenty > >>>>>>> of > >>>>>>>> disk, CPU, and RAM free. It's running a recent version of MAPR > >>>>> Hadoop > >>>>>> on > >>>>>>>> CentOS 6.5 on EC2. It feels like something is hanging > >>>>>>>> > >>>>>>>> Note: This is just after starting sqlline; no actual work, which > >>>>> might > >>>>>>> have > >>>>>>>> slowed things down, has happened yet. > >>>>>>>> > >>>>>>>> This is what's running from a Drill perspective: > >>>>>>>> > >>>>>>>> ps aux | grep drill > >>>>>>>> ec2-user 24662 0.0 0.0 106100 1328 pts/0 S 12:29 0:00 > >>>> bash > >>>>>>>> bin/drillbit.sh internal_start drillbit > >>>>>>>> ec2-user 24697 0.3 2.3 6486240 730568 pts/0 Sl 12:29 0:36 > >>>>>>>> /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79.x86_64/bin/java > >>>>>>>> -Dlog.path=/opt/mapr/drill/drill-0.9.0/logs/drillbit.log -Xms1G > >>>>> -Xmx4G > >>>>>>>> -XX:MaxDirectMemorySize=8G -XX:MaxPermSize=512M > >>>>>>>> -XX:ReservedCodeCacheSize=1G -ea > >>>>>>>> -Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf > >>>>>>>> -Dzookeeper.sasl.client=false -XX:+CMSClassUnloadingEnabled > >>>>>>>> -XX:+UseConcMarkSweepGC -cp > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >> > /opt/mapr/drill/drill-0.9.0/conf:/opt/mapr/drill/drill-0.9.0/jars/*:/opt/mapr/drill/drill-0.9.0/jars/ext/*:/opt/mapr/drill/drill-0.9.0/jars/3rdparty/*:/opt/mapr/drill/drill-0.9.0/jars/classb/* > >>>>>>>> org.apache.drill.exec.server.Drillbit > >>>>>>>> > >>>>>>>> Drill has been started thusly: > >>>>>>>> > >>>>>>>> bin/sqlline -u jdbc:drill:zk=theServer:5181 > >>>>>>>> > >>>>>>>> Any idea how I could troubleshoot this? > >>>>>>>> > >>>>>>>> Anyone else experiencing slowness after upgrading? > >>>>>>>> > >>>>>>>> Thank you > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> > >>>>> Abdelhakim Deneche > >>>>> > >>>>> Software Engineer > >>>>> > >>>>> <http://www.mapr.com/> > >>>>> > >>>>> > >>>>> Now Available - Free Hadoop On-Demand Training > >>>>> < > >>>>> > >>>> > >> > http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available > >>>>>> > >>>>> > >>>> > >> > >> > >
