instead of file:/// use maprfs:/// (I assume it is a mapr cluster) If you do that you don’t need to use /mapr/clustername in front of the location path, but you can rather just directly point to the maprfs directory.
See the sample I posted vs the storage plug in you had configured. The default DFS plugin should also show you the correct config info. —Andries On May 11, 2015, at 5:21 PM, Minnow Noir <[email protected]> wrote: > I'm not sure I understand what you mean, Andries. What field and value are > you suggesting that I change? If there's a better way to do this, I would > like to do it the right way, even if it's not the issue in question. > > Thanks > > On Mon, May 11, 2015 at 8:14 PM, Andries Engelbrecht < > [email protected]> wrote: > >> You set up the Storage plugin to use the loopback NFS to read the file, >> which is not the best way to do it. >> >> Why not use the MapR-FS or HDFS plug ins , this is a setup that uses the >> MapR-FS plugin which will be considerably better than the loopback NFS for >> Drill >> >> >>> { >>> "type": "file", >>> "enabled": true, >>> "connection": “maprfs:///", >>> "workspaces": { >>> "schema1": { >>> "location": "/user/theUser/test", >>> "writable": true, >>> "defaultInputFormat": null >>> } >>> }, >> >> >> —Andries >> >> >> On May 11, 2015, at 5:07 PM, Minnow Noir <[email protected]> wrote: >> >>> Hanifi: The web UI is extremely slow, and the style sheets aren't >> loading, >>> but here is what I see for "show databases" >>> >>> Major FragmentMinor Fragments ReportingFirst StartLast StartFirst EndLast >>> Endtmintavgtmaxlast updatelast progressmemmax 00-xx-xx1 / >>> 10.213s0.213s0.670s0.670s0.457s0.457s0.457s12:15:3012:15:303MB >>> Major Fragment: 00-xx-xx >>> < >> http://174.129.247.50:8047/profiles/2aaf002d-0351-e983-b9a1-51da33885d87#fragment-0 >>> >>> Minor FragmentHostStartEndTotal TimeMax RecordsMax BatchesLast >> UpdateLast >>> ProgressPeak MemoryState >> 00-00-xxip-10-101-197-197.ec2.internal0.213s0.670s >>> 0.457s6112:15:3012:15:303MBFINISHED >>> Operator Profiles >>> Overview >>> < >> http://174.129.247.50:8047/profiles/2aaf002d-0351-e983-b9a1-51da33885d87#operator-overview >>> >>> OperatorTypeSetup (min)Setup (avg)Setup (max)Process (min)Process >>> (avg)Process >>> (max)Wait (min)Wait (avg)Wait (max)Mem (avg)Mem (max) >> 00-xx-00SCREEN0.000s >>> 0.000s0.000s0.000s0.000s0.000s0.000s0.000s0.000s-- 00-xx-01PROJECT0.422s >>> 0.422s0.422s0.001s0.001s0.001s0.000s0.000s0.000s-- 00-xx-02 >>> >> INFO_SCHEMA_SUB_SCAN0.000s0.000s0.000s0.020s0.020s0.020s0.000s0.000s0.000s >>> 17MB17MB >>> 00-xx-00 - SCREEN >>> < >> http://174.129.247.50:8047/profiles/2aaf002d-0351-e983-b9a1-51da33885d87#operator-0-0 >>> >>> Minor FragmentSetupProcessWaitMax BatchesMax RecordsPeak Mem 00-00-00 >>> 0.000s0.000s0.000s16- >>> 00-xx-01 - PROJECT >>> < >> http://174.129.247.50:8047/profiles/2aaf002d-0351-e983-b9a1-51da33885d87#operator-0-1 >>> >>> Minor FragmentSetupProcessWaitMax BatchesMax RecordsPeak Mem 00-00-01 >>> 0.422s0.001s0.000s16- >>> 00-xx-02 - INFO_SCHEMA_SUB_SCAN >>> < >> http://174.129.247.50:8047/profiles/2aaf002d-0351-e983-b9a1-51da33885d87#operator-0-2 >>> >>> Minor FragmentSetupProcessWaitMax BatchesMax RecordsPeak Mem >>> 00-00-020.000s0.020s0.000s0017MB >>> >>> Andries: That storage plugin is very simple. I've obfuscated some >>> proprietary info. (Note that the web UI was slow/hanging before I added >>> the storage plugin.) >>> >>> { >>> "type": "file", >>> "enabled": true, >>> "connection": "file:///", >>> "workspaces": { >>> "schema1": { >>> "location": "/mapr/theClusterName/user/theUser/test", >>> "writable": true, >>> "defaultInputFormat": null >>> } >>> }, >>> "formats": { >>> "csv": { >>> "type": "text", >>> "extensions": [ >>> "csv" >>> ], >>> "delimiter": "," >>> }, >>> "tsv": { >>> "type": "text", >>> "extensions": [ >>> "tsv" >>> ], >>> "delimiter": "\t" >>> }, >>> "parquet": { >>> "type": "parquet" >>> } >>> } >>> } >>> >>> Unfortunately, because the Web UI is having issues, it is presenting an >>> unstyled page with no disable links for the storage plugin so I can't >>> disable it. With the web UI not working, I don't know how to delete the >>> plugin. >>> >>> Abdel: There are no errors in the logs. >>> >>> >>> On Mon, May 11, 2015 at 7:22 PM, Hanifi Gunes <[email protected]> >> wrote: >>> >>>> I would be interested in knowing where the time has been spent. Can you >>>> inspect query profile from web ui for `show databases` and let us know >> how >>>> the profile looks like? >>>> >>>> On Mon, May 11, 2015 at 4:10 PM, Abdel Hakim Deneche < >>>> [email protected]> >>>> wrote: >>>> >>>>> any errors in the logs ? >>>>> >>>>> On Mon, May 11, 2015 at 4:03 PM, Minnow Noir <[email protected]> >>>> wrote: >>>>> >>>>>> Yes. The bit and server are idle, with no jobs of any sort running on >>>>>> them. >>>>>> >>>>>> I just did started sqlline and it happened again. I had to cancel the >>>>> show >>>>>> databases command after 30s. >>>>>> >>>>>> sqlline version 1.1.6 >>>>>> 0: jdbc:drill:zk=theServer:5181> show databases; >>>>>> +-------------+ >>>>>> | SCHEMA_NAME | >>>>>> +-------------+ >>>>>> | INFORMATION_SCHEMA | >>>>>> | cp.default | >>>>>> | dfs.default | >>>>>> | dfs.root | >>>>>> | dfs.tmp | >>>>>> | test.default | >>>>>> | test.schema1 | >>>>>> | sys | >>>>>> +-------------+ >>>>>> 8 rows selected (29.264 seconds) >>>>>> >>>>>> On Mon, May 11, 2015 at 6:57 PM, Hanifi Gunes <[email protected]> >>>>> wrote: >>>>>> >>>>>>> Can you confirm if this happens even when the particular 0.9 bit is >>>>> idle? >>>>>>> >>>>>>> -Hanifi >>>>>>> >>>>>>> On Mon, May 11, 2015 at 3:53 PM, Minnow Noir <[email protected]> >>>>>> wrote: >>>>>>> >>>>>>>> Experiencing some slowness after upgrading one of our servers from >>>>> 0.8 >>>>>> to >>>>>>>> 0.9. In particular, despite restarting the Drillbit, and even the >>>>>>> server, >>>>>>>> several times, the Drill web UI and sqlline are both so slow as to >>>> be >>>>>>>> unusable. >>>>>>>> >>>>>>>> For example, doing a show databases hangs indefinitely, until I >>>>> CTRL-C >>>>>>> the >>>>>>>> command (e.g., after a minute). >>>>>>>> >>>>>>>> show databases; >>>>>>>> +-------------+ >>>>>>>> | SCHEMA_NAME | >>>>>>>> +-------------+ >>>>>>>> | INFORMATION_SCHEMA | >>>>>>>> | cp.default | >>>>>>>> | dfs.default | >>>>>>>> | dfs.root | >>>>>>>> | dfs.tmp | >>>>>>>> | test.default | >>>>>>>> | test.schema1 | >>>>>>>> | sys | >>>>>>>> +-------------+ >>>>>>>> 8 rows selected (59.044 seconds) >>>>>>>> >>>>>>>> I tried changing schemas, but had to cancel after waiting several >>>>>>> minutes. >>>>>>>> >>>>>>>> use test.schema1; >>>>>>>> +------------+------------+ >>>>>>>> | ok | summary | >>>>>>>> +------------+------------+ >>>>>>>> | true | Default schema changed to 'test.schema1' | >>>>>>>> +------------+------------+ >>>>>>>> 1 row selected (208.089 seconds) >>>>>>>> >>>>>>>> There are no errors or obvious issues in the logs. The server has >>>>>> plenty >>>>>>> of >>>>>>>> disk, CPU, and RAM free. It's running a recent version of MAPR >>>>> Hadoop >>>>>> on >>>>>>>> CentOS 6.5 on EC2. It feels like something is hanging >>>>>>>> >>>>>>>> Note: This is just after starting sqlline; no actual work, which >>>>> might >>>>>>> have >>>>>>>> slowed things down, has happened yet. >>>>>>>> >>>>>>>> This is what's running from a Drill perspective: >>>>>>>> >>>>>>>> ps aux | grep drill >>>>>>>> ec2-user 24662 0.0 0.0 106100 1328 pts/0 S 12:29 0:00 >>>> bash >>>>>>>> bin/drillbit.sh internal_start drillbit >>>>>>>> ec2-user 24697 0.3 2.3 6486240 730568 pts/0 Sl 12:29 0:36 >>>>>>>> /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79.x86_64/bin/java >>>>>>>> -Dlog.path=/opt/mapr/drill/drill-0.9.0/logs/drillbit.log -Xms1G >>>>> -Xmx4G >>>>>>>> -XX:MaxDirectMemorySize=8G -XX:MaxPermSize=512M >>>>>>>> -XX:ReservedCodeCacheSize=1G -ea >>>>>>>> -Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf >>>>>>>> -Dzookeeper.sasl.client=false -XX:+CMSClassUnloadingEnabled >>>>>>>> -XX:+UseConcMarkSweepGC -cp >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >> /opt/mapr/drill/drill-0.9.0/conf:/opt/mapr/drill/drill-0.9.0/jars/*:/opt/mapr/drill/drill-0.9.0/jars/ext/*:/opt/mapr/drill/drill-0.9.0/jars/3rdparty/*:/opt/mapr/drill/drill-0.9.0/jars/classb/* >>>>>>>> org.apache.drill.exec.server.Drillbit >>>>>>>> >>>>>>>> Drill has been started thusly: >>>>>>>> >>>>>>>> bin/sqlline -u jdbc:drill:zk=theServer:5181 >>>>>>>> >>>>>>>> Any idea how I could troubleshoot this? >>>>>>>> >>>>>>>> Anyone else experiencing slowness after upgrading? >>>>>>>> >>>>>>>> Thank you >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Abdelhakim Deneche >>>>> >>>>> Software Engineer >>>>> >>>>> <http://www.mapr.com/> >>>>> >>>>> >>>>> Now Available - Free Hadoop On-Demand Training >>>>> < >>>>> >>>> >> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available >>>>>> >>>>> >>>> >> >>
