Hanifi: The web UI is extremely slow, and the style sheets aren't loading, but here is what I see for "show databases"
Major FragmentMinor Fragments ReportingFirst StartLast StartFirst EndLast Endtmintavgtmaxlast updatelast progressmemmax 00-xx-xx1 / 10.213s0.213s0.670s0.670s0.457s0.457s0.457s12:15:3012:15:303MB Major Fragment: 00-xx-xx <http://174.129.247.50:8047/profiles/2aaf002d-0351-e983-b9a1-51da33885d87#fragment-0> Minor FragmentHostStartEndTotal TimeMax RecordsMax BatchesLast UpdateLast ProgressPeak MemoryState 00-00-xxip-10-101-197-197.ec2.internal0.213s0.670s 0.457s6112:15:3012:15:303MBFINISHED Operator Profiles Overview <http://174.129.247.50:8047/profiles/2aaf002d-0351-e983-b9a1-51da33885d87#operator-overview> OperatorTypeSetup (min)Setup (avg)Setup (max)Process (min)Process (avg)Process (max)Wait (min)Wait (avg)Wait (max)Mem (avg)Mem (max) 00-xx-00SCREEN0.000s 0.000s0.000s0.000s0.000s0.000s0.000s0.000s0.000s-- 00-xx-01PROJECT0.422s 0.422s0.422s0.001s0.001s0.001s0.000s0.000s0.000s-- 00-xx-02 INFO_SCHEMA_SUB_SCAN0.000s0.000s0.000s0.020s0.020s0.020s0.000s0.000s0.000s 17MB17MB 00-xx-00 - SCREEN <http://174.129.247.50:8047/profiles/2aaf002d-0351-e983-b9a1-51da33885d87#operator-0-0> Minor FragmentSetupProcessWaitMax BatchesMax RecordsPeak Mem 00-00-00 0.000s0.000s0.000s16- 00-xx-01 - PROJECT <http://174.129.247.50:8047/profiles/2aaf002d-0351-e983-b9a1-51da33885d87#operator-0-1> Minor FragmentSetupProcessWaitMax BatchesMax RecordsPeak Mem 00-00-01 0.422s0.001s0.000s16- 00-xx-02 - INFO_SCHEMA_SUB_SCAN <http://174.129.247.50:8047/profiles/2aaf002d-0351-e983-b9a1-51da33885d87#operator-0-2> Minor FragmentSetupProcessWaitMax BatchesMax RecordsPeak Mem 00-00-020.000s0.020s0.000s0017MB Andries: That storage plugin is very simple. I've obfuscated some proprietary info. (Note that the web UI was slow/hanging before I added the storage plugin.) { "type": "file", "enabled": true, "connection": "file:///", "workspaces": { "schema1": { "location": "/mapr/theClusterName/user/theUser/test", "writable": true, "defaultInputFormat": null } }, "formats": { "csv": { "type": "text", "extensions": [ "csv" ], "delimiter": "," }, "tsv": { "type": "text", "extensions": [ "tsv" ], "delimiter": "\t" }, "parquet": { "type": "parquet" } } } Unfortunately, because the Web UI is having issues, it is presenting an unstyled page with no disable links for the storage plugin so I can't disable it. With the web UI not working, I don't know how to delete the plugin. Abdel: There are no errors in the logs. On Mon, May 11, 2015 at 7:22 PM, Hanifi Gunes <[email protected]> wrote: > I would be interested in knowing where the time has been spent. Can you > inspect query profile from web ui for `show databases` and let us know how > the profile looks like? > > On Mon, May 11, 2015 at 4:10 PM, Abdel Hakim Deneche < > [email protected]> > wrote: > > > any errors in the logs ? > > > > On Mon, May 11, 2015 at 4:03 PM, Minnow Noir <[email protected]> > wrote: > > > > > Yes. The bit and server are idle, with no jobs of any sort running on > > > them. > > > > > > I just did started sqlline and it happened again. I had to cancel the > > show > > > databases command after 30s. > > > > > > sqlline version 1.1.6 > > > 0: jdbc:drill:zk=theServer:5181> show databases; > > > +-------------+ > > > | SCHEMA_NAME | > > > +-------------+ > > > | INFORMATION_SCHEMA | > > > | cp.default | > > > | dfs.default | > > > | dfs.root | > > > | dfs.tmp | > > > | test.default | > > > | test.schema1 | > > > | sys | > > > +-------------+ > > > 8 rows selected (29.264 seconds) > > > > > > On Mon, May 11, 2015 at 6:57 PM, Hanifi Gunes <[email protected]> > > wrote: > > > > > > > Can you confirm if this happens even when the particular 0.9 bit is > > idle? > > > > > > > > -Hanifi > > > > > > > > On Mon, May 11, 2015 at 3:53 PM, Minnow Noir <[email protected]> > > > wrote: > > > > > > > > > Experiencing some slowness after upgrading one of our servers from > > 0.8 > > > to > > > > > 0.9. In particular, despite restarting the Drillbit, and even the > > > > server, > > > > > several times, the Drill web UI and sqlline are both so slow as to > be > > > > > unusable. > > > > > > > > > > For example, doing a show databases hangs indefinitely, until I > > CTRL-C > > > > the > > > > > command (e.g., after a minute). > > > > > > > > > > show databases; > > > > > +-------------+ > > > > > | SCHEMA_NAME | > > > > > +-------------+ > > > > > | INFORMATION_SCHEMA | > > > > > | cp.default | > > > > > | dfs.default | > > > > > | dfs.root | > > > > > | dfs.tmp | > > > > > | test.default | > > > > > | test.schema1 | > > > > > | sys | > > > > > +-------------+ > > > > > 8 rows selected (59.044 seconds) > > > > > > > > > > I tried changing schemas, but had to cancel after waiting several > > > > minutes. > > > > > > > > > > use test.schema1; > > > > > +------------+------------+ > > > > > | ok | summary | > > > > > +------------+------------+ > > > > > | true | Default schema changed to 'test.schema1' | > > > > > +------------+------------+ > > > > > 1 row selected (208.089 seconds) > > > > > > > > > > There are no errors or obvious issues in the logs. The server has > > > plenty > > > > of > > > > > disk, CPU, and RAM free. It's running a recent version of MAPR > > Hadoop > > > on > > > > > CentOS 6.5 on EC2. It feels like something is hanging > > > > > > > > > > Note: This is just after starting sqlline; no actual work, which > > might > > > > have > > > > > slowed things down, has happened yet. > > > > > > > > > > This is what's running from a Drill perspective: > > > > > > > > > > ps aux | grep drill > > > > > ec2-user 24662 0.0 0.0 106100 1328 pts/0 S 12:29 0:00 > bash > > > > > bin/drillbit.sh internal_start drillbit > > > > > ec2-user 24697 0.3 2.3 6486240 730568 pts/0 Sl 12:29 0:36 > > > > > /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79.x86_64/bin/java > > > > > -Dlog.path=/opt/mapr/drill/drill-0.9.0/logs/drillbit.log -Xms1G > > -Xmx4G > > > > > -XX:MaxDirectMemorySize=8G -XX:MaxPermSize=512M > > > > > -XX:ReservedCodeCacheSize=1G -ea > > > > > -Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf > > > > > -Dzookeeper.sasl.client=false -XX:+CMSClassUnloadingEnabled > > > > > -XX:+UseConcMarkSweepGC -cp > > > > > > > > > > > > > > > > > > > > /opt/mapr/drill/drill-0.9.0/conf:/opt/mapr/drill/drill-0.9.0/jars/*:/opt/mapr/drill/drill-0.9.0/jars/ext/*:/opt/mapr/drill/drill-0.9.0/jars/3rdparty/*:/opt/mapr/drill/drill-0.9.0/jars/classb/* > > > > > org.apache.drill.exec.server.Drillbit > > > > > > > > > > Drill has been started thusly: > > > > > > > > > > bin/sqlline -u jdbc:drill:zk=theServer:5181 > > > > > > > > > > Any idea how I could troubleshoot this? > > > > > > > > > > Anyone else experiencing slowness after upgrading? > > > > > > > > > > Thank you > > > > > > > > > > > > > > > > > > > > -- > > > > Abdelhakim Deneche > > > > Software Engineer > > > > <http://www.mapr.com/> > > > > > > Now Available - Free Hadoop On-Demand Training > > < > > > http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available > > > > > >
