You set up the Storage plugin to use the loopback NFS to read the file, which
is not the best way to do it.
Why not use the MapR-FS or HDFS plug ins , this is a setup that uses the
MapR-FS plugin which will be considerably better than the loopback NFS for Drill
> {
> "type": "file",
> "enabled": true,
> "connection": “maprfs:///",
> "workspaces": {
> "schema1": {
> "location": "/user/theUser/test",
> "writable": true,
> "defaultInputFormat": null
> }
> },
—Andries
On May 11, 2015, at 5:07 PM, Minnow Noir <[email protected]> wrote:
> Hanifi: The web UI is extremely slow, and the style sheets aren't loading,
> but here is what I see for "show databases"
>
> Major FragmentMinor Fragments ReportingFirst StartLast StartFirst EndLast
> Endtmintavgtmaxlast updatelast progressmemmax 00-xx-xx1 /
> 10.213s0.213s0.670s0.670s0.457s0.457s0.457s12:15:3012:15:303MB
> Major Fragment: 00-xx-xx
> <http://174.129.247.50:8047/profiles/2aaf002d-0351-e983-b9a1-51da33885d87#fragment-0>
> Minor FragmentHostStartEndTotal TimeMax RecordsMax BatchesLast UpdateLast
> ProgressPeak MemoryState 00-00-xxip-10-101-197-197.ec2.internal0.213s0.670s
> 0.457s6112:15:3012:15:303MBFINISHED
> Operator Profiles
> Overview
> <http://174.129.247.50:8047/profiles/2aaf002d-0351-e983-b9a1-51da33885d87#operator-overview>
> OperatorTypeSetup (min)Setup (avg)Setup (max)Process (min)Process
> (avg)Process
> (max)Wait (min)Wait (avg)Wait (max)Mem (avg)Mem (max) 00-xx-00SCREEN0.000s
> 0.000s0.000s0.000s0.000s0.000s0.000s0.000s0.000s-- 00-xx-01PROJECT0.422s
> 0.422s0.422s0.001s0.001s0.001s0.000s0.000s0.000s-- 00-xx-02
> INFO_SCHEMA_SUB_SCAN0.000s0.000s0.000s0.020s0.020s0.020s0.000s0.000s0.000s
> 17MB17MB
> 00-xx-00 - SCREEN
> <http://174.129.247.50:8047/profiles/2aaf002d-0351-e983-b9a1-51da33885d87#operator-0-0>
> Minor FragmentSetupProcessWaitMax BatchesMax RecordsPeak Mem 00-00-00
> 0.000s0.000s0.000s16-
> 00-xx-01 - PROJECT
> <http://174.129.247.50:8047/profiles/2aaf002d-0351-e983-b9a1-51da33885d87#operator-0-1>
> Minor FragmentSetupProcessWaitMax BatchesMax RecordsPeak Mem 00-00-01
> 0.422s0.001s0.000s16-
> 00-xx-02 - INFO_SCHEMA_SUB_SCAN
> <http://174.129.247.50:8047/profiles/2aaf002d-0351-e983-b9a1-51da33885d87#operator-0-2>
> Minor FragmentSetupProcessWaitMax BatchesMax RecordsPeak Mem
> 00-00-020.000s0.020s0.000s0017MB
>
> Andries: That storage plugin is very simple. I've obfuscated some
> proprietary info. (Note that the web UI was slow/hanging before I added
> the storage plugin.)
>
> {
> "type": "file",
> "enabled": true,
> "connection": "file:///",
> "workspaces": {
> "schema1": {
> "location": "/mapr/theClusterName/user/theUser/test",
> "writable": true,
> "defaultInputFormat": null
> }
> },
> "formats": {
> "csv": {
> "type": "text",
> "extensions": [
> "csv"
> ],
> "delimiter": ","
> },
> "tsv": {
> "type": "text",
> "extensions": [
> "tsv"
> ],
> "delimiter": "\t"
> },
> "parquet": {
> "type": "parquet"
> }
> }
> }
>
> Unfortunately, because the Web UI is having issues, it is presenting an
> unstyled page with no disable links for the storage plugin so I can't
> disable it. With the web UI not working, I don't know how to delete the
> plugin.
>
> Abdel: There are no errors in the logs.
>
>
> On Mon, May 11, 2015 at 7:22 PM, Hanifi Gunes <[email protected]> wrote:
>
>> I would be interested in knowing where the time has been spent. Can you
>> inspect query profile from web ui for `show databases` and let us know how
>> the profile looks like?
>>
>> On Mon, May 11, 2015 at 4:10 PM, Abdel Hakim Deneche <
>> [email protected]>
>> wrote:
>>
>>> any errors in the logs ?
>>>
>>> On Mon, May 11, 2015 at 4:03 PM, Minnow Noir <[email protected]>
>> wrote:
>>>
>>>> Yes. The bit and server are idle, with no jobs of any sort running on
>>>> them.
>>>>
>>>> I just did started sqlline and it happened again. I had to cancel the
>>> show
>>>> databases command after 30s.
>>>>
>>>> sqlline version 1.1.6
>>>> 0: jdbc:drill:zk=theServer:5181> show databases;
>>>> +-------------+
>>>> | SCHEMA_NAME |
>>>> +-------------+
>>>> | INFORMATION_SCHEMA |
>>>> | cp.default |
>>>> | dfs.default |
>>>> | dfs.root |
>>>> | dfs.tmp |
>>>> | test.default |
>>>> | test.schema1 |
>>>> | sys |
>>>> +-------------+
>>>> 8 rows selected (29.264 seconds)
>>>>
>>>> On Mon, May 11, 2015 at 6:57 PM, Hanifi Gunes <[email protected]>
>>> wrote:
>>>>
>>>>> Can you confirm if this happens even when the particular 0.9 bit is
>>> idle?
>>>>>
>>>>> -Hanifi
>>>>>
>>>>> On Mon, May 11, 2015 at 3:53 PM, Minnow Noir <[email protected]>
>>>> wrote:
>>>>>
>>>>>> Experiencing some slowness after upgrading one of our servers from
>>> 0.8
>>>> to
>>>>>> 0.9. In particular, despite restarting the Drillbit, and even the
>>>>> server,
>>>>>> several times, the Drill web UI and sqlline are both so slow as to
>> be
>>>>>> unusable.
>>>>>>
>>>>>> For example, doing a show databases hangs indefinitely, until I
>>> CTRL-C
>>>>> the
>>>>>> command (e.g., after a minute).
>>>>>>
>>>>>> show databases;
>>>>>> +-------------+
>>>>>> | SCHEMA_NAME |
>>>>>> +-------------+
>>>>>> | INFORMATION_SCHEMA |
>>>>>> | cp.default |
>>>>>> | dfs.default |
>>>>>> | dfs.root |
>>>>>> | dfs.tmp |
>>>>>> | test.default |
>>>>>> | test.schema1 |
>>>>>> | sys |
>>>>>> +-------------+
>>>>>> 8 rows selected (59.044 seconds)
>>>>>>
>>>>>> I tried changing schemas, but had to cancel after waiting several
>>>>> minutes.
>>>>>>
>>>>>> use test.schema1;
>>>>>> +------------+------------+
>>>>>> | ok | summary |
>>>>>> +------------+------------+
>>>>>> | true | Default schema changed to 'test.schema1' |
>>>>>> +------------+------------+
>>>>>> 1 row selected (208.089 seconds)
>>>>>>
>>>>>> There are no errors or obvious issues in the logs. The server has
>>>> plenty
>>>>> of
>>>>>> disk, CPU, and RAM free. It's running a recent version of MAPR
>>> Hadoop
>>>> on
>>>>>> CentOS 6.5 on EC2. It feels like something is hanging
>>>>>>
>>>>>> Note: This is just after starting sqlline; no actual work, which
>>> might
>>>>> have
>>>>>> slowed things down, has happened yet.
>>>>>>
>>>>>> This is what's running from a Drill perspective:
>>>>>>
>>>>>> ps aux | grep drill
>>>>>> ec2-user 24662 0.0 0.0 106100 1328 pts/0 S 12:29 0:00
>> bash
>>>>>> bin/drillbit.sh internal_start drillbit
>>>>>> ec2-user 24697 0.3 2.3 6486240 730568 pts/0 Sl 12:29 0:36
>>>>>> /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79.x86_64/bin/java
>>>>>> -Dlog.path=/opt/mapr/drill/drill-0.9.0/logs/drillbit.log -Xms1G
>>> -Xmx4G
>>>>>> -XX:MaxDirectMemorySize=8G -XX:MaxPermSize=512M
>>>>>> -XX:ReservedCodeCacheSize=1G -ea
>>>>>> -Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf
>>>>>> -Dzookeeper.sasl.client=false -XX:+CMSClassUnloadingEnabled
>>>>>> -XX:+UseConcMarkSweepGC -cp
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> /opt/mapr/drill/drill-0.9.0/conf:/opt/mapr/drill/drill-0.9.0/jars/*:/opt/mapr/drill/drill-0.9.0/jars/ext/*:/opt/mapr/drill/drill-0.9.0/jars/3rdparty/*:/opt/mapr/drill/drill-0.9.0/jars/classb/*
>>>>>> org.apache.drill.exec.server.Drillbit
>>>>>>
>>>>>> Drill has been started thusly:
>>>>>>
>>>>>> bin/sqlline -u jdbc:drill:zk=theServer:5181
>>>>>>
>>>>>> Any idea how I could troubleshoot this?
>>>>>>
>>>>>> Anyone else experiencing slowness after upgrading?
>>>>>>
>>>>>> Thank you
>>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Abdelhakim Deneche
>>>
>>> Software Engineer
>>>
>>> <http://www.mapr.com/>
>>>
>>>
>>> Now Available - Free Hadoop On-Demand Training
>>> <
>>>
>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>>>>
>>>
>>
