J-D, Thank you very much for you reminding. It may be my fault, the node-5 is a new added machine, I forgot to modify the "ulimit" of max number of open files. In fact, in datanode-5's logs, there are "Too many open files" exception. 3 months ago, I had modified it on other nodes. I can remember this FAQ of HBase: http://wiki.apache.org/hadoop/Hbase/FAQ(6).
I have changed the nofile to 32768 now. I will restart the cluster and retest. I will temporarily reply this note to some of other messages from me, to avoid confusion of other community member. Thank you so much and so sorry. Now, it is 4 o'clock in the morning, I want to get a sleep :-). Schubert On Sat, Mar 14, 2009 at 3:34 AM, Jean-Daniel Cryans <[email protected]>wrote: > Schubert, > > HW sounds good enough. > > On the "Could not complete write to file", isn't there are "caused by" > clause following it? Hard to tell anything from these excerpts. Can > you grep your logs to see any occurrence of "Too many open files" too? > > Finally, if you could drop by on the IRC channel it would be great, we > will help you digging in the logs. > > J-D > > On Fri, Mar 13, 2009 at 3:25 PM, schubert zhang <[email protected]> wrote: > > Hi J-D, > > Thanks for your guide. > > > > Following is my cluster and hardware information: > > > > Node0: HDFS Namenode, HBase Master. > > Node1-5: HDFS Datanode, HBase RegionServer > > > > DELL 2950 server, each node have 4GB memory. > > Node0-4 have 1TB disk (0.8TB for HDFS) > > Node5 have 1.5TB disk (1.2TB for HDFS) > > > > I am using hadoop-0.19.1 and hbase-0.19.0. > > > > My table schema is: > > {NAME => 'TESTTAB', IS_ROOT => 'false', IS_META => 'false', FAMILIES => > > [{NAME => 'cdr', BLOOMFILTER => 'false' > > , VERSIONS => '1', COMPRESSION => 'BLOCK', LENGTH => '2147483647', TTL => > > '-1', IN_MEMORY => 'false', BLOCKCAC > > HE => 'true'}], INDEXES => []} > > I am using BLOCKCACHE and COMPRESSION now. I am also waiting to > > use BLOOMFILTER in the future (0.20.x) > > > > My region size is 128MB. > > > > At about 70 000 000 rows (each row has about 400B raw data), there are > about > > 125 regions. And in HDFS, there is only 16GB (by "hadoop fs -dus > > /hbase/TESTTAB') used. On the HDFS web GUI, there is only 1.x% storage > used. > > I think this volume is small for my hardware and storage. > > > > I checked the logs of HDFS, there are some exceptions, but it is so hard > > to associate these issues. I will check more of HDFS and try the > > HBase-0.19.1 RC1. Thanks. > > > > In namenode, there are some exceptions like: > > .16_1236922775003_60020/hlog.dat.1236925456649 because > dir.getFileBlocks() > > is null and pendingFile is null > > 2009-03-13 14:33:03,828 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 8 on 9000, call > > > complete(/hbase/log_10.24.1.16_1236922775003_60020/hlog.dat.1236925456649, > > DFSClient_-1142813574) from 10.24.1.16:58444: error: > java.io.IOException: > > Could not complete write to file > > /hbase/log_10.24.1.16_1236922775003_60020/hlog.dat.1236925456649 by > > DFSClient_-1142813574 > > java.io.IOException: Could not complete write to file > > /hbase/log_10.24.1.16_1236922775003_60020/hlog.dat.1236925456649 by > > DFSClient_-1142813574 > > at > > > org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:378) > > at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481) > > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894) > > > > In datanode, like: > > 2009-03-13 14:28:45,519 WARN > > org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration( > > 10.24.1.12:50010, > storageID=DS-1393238225-10.24.1.12-50010-1236857766916, > > infoPort=50075, ipcPort=50020):Got exception while serving > > blk_-4945343177244655889_64091 to /10.24.1.10: > > java.io.IOException: Block blk_-4945343177244655889_64091 is not valid. > > at > > > org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:726) > > at > > > org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:714) > > at > > > org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:92) > > at > > > org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:172) > > at > > > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:95) > > at java.lang.Thread.run(Thread.java:619) > > > > 2009-03-13 14:28:45,519 ERROR > > org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration( > > 10.24.1.12:50010, > storageID=DS-1393238225-10.24.1.12-50010-1236857766916, > > infoPort=50075, ipcPort=50020):DataXceiver > > java.io.IOException: Block blk_-4945343177244655889_64091 is not valid. > > at > > > org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:726) > > at > > > org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:714) > > at > > > org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:92) > > at > > > org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:172) > > at > > > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:95) > > at java.lang.Thread.run(Thread.java:619) > > > > Schubert > > > > On Fri, Mar 13, 2009 at 10:31 PM, Jean-Daniel Cryans < > [email protected]>wrote: > > > >> schubert, > >> > >> Yeah at 70 000 000 rows with only 5 region servers you might expect > >> stuff like that... But this is not good. Can you take a look in your > >> datanodes logs to see if there are any obvious exceptions around the > >> time of the "error closing and deleting..." exception? > >> > >> I'd like to know more about your hardware. > >> > >> Also I strongly suggest that you try out the 0.19.1 RC1, it has really > >> nice fixes. > >> > >> Thx for reporting all that stuff, > >> > >> J-D > >> > >> On Fri, Mar 13, 2009 at 5:41 AM, schubert zhang <[email protected]> > wrote: > >> > I have a 5+1 HBase/Hadoop cluster. (5 region server and 1 master)A > table > >> > TESTTAB with only one column family with 36 qualifiers. > >> > A process in the master node use batchUpdate (autoFlush=false) to > insert > >> > random rows into this table. > >> > After about 70,000,000 rows inserted, failure. And on the web GUI of > >> HBase, > >> > I can only find 4 regionservers. Then, I ssh into the missed node to > >> check > >> > the log. > >> > > >> > There is a very big log file (12GB), and when I "tail -f .." this log > >> file, > >> > endless FileNotFoundException is printing, like following: > >> > > >> > 2009-03-13 14:32:21,617 WARN > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: error getting > store > >> file > >> > index size for 310680591/cdr: java.io.FileNotFoundException: File does > >> not > >> > exist: > >> > > >> > hdfs://nd0-rack0-cloud:9000/hbase/TESTTAB/310680591/cdr/mapfiles/1049684492857034443/index > >> > at > >> > > >> > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:415) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HStoreFile.indexLength(HStoreFile.java:488) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HStore.getStorefilesIndexSize(HStore.java:2179) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.doMetrics(HRegionServer.java:941) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:334) > >> > at java.lang.Thread.run(Thread.java:619) 2009-03-13 14:32:21,627 WARN > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message > >> > (Retry: 0) java.io.FileNotFoundException: File does not exist: > >> > > >> > hdfs://nd0-rack0-cloud:9000/hbase/TESTTAB/310680591/cdr/mapfiles/1049684492857034443/index > >> > at > >> > > >> > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:415) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HStoreFile.indexLength(HStoreFile.java:488) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HStore.getStorefilesIndexSize(HStore.java:2179) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.createRegionLoad(HRegionServer.java:625) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:341) > >> > at java.lang.Thread.run(Thread.java:619) 2009-03-13 14:32:21,628 WARN > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: error getting > store > >> file > >> > index size for 310680591/cdr: java.io.FileNotFoundException: File does > >> not > >> > exist: > >> > > >> > hdfs://nd0-rack0-cloud:9000/hbase/TESTTAB/310680591/cdr/mapfiles/1049684492857034443/index > >> > at > >> > > >> > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:415) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HStoreFile.indexLength(HStoreFile.java:488) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HStore.getStorefilesIndexSize(HStore.java:2179) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.doMetrics(HRegionServer.java:941) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:334) > >> > at java.lang.Thread.run(Thread.java:619) 2009-03-13 14:32:21,629 WARN > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message > >> > (Retry: 1) java.io.FileNotFoundException: File does not exist: > >> > > >> > hdfs://nd0-rack0-cloud:9000/hbase/TESTTAB/310680591/cdr/mapfiles/1049684492857034443/index > >> > at > >> > > >> > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:415) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HStoreFile.indexLength(HStoreFile.java:488) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HStore.getStorefilesIndexSize(HStore.java:2179) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.createRegionLoad(HRegionServer.java:625) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:341) > >> > at java.lang.Thread.run(Thread.java:619) > >> > > >> > ....... > >> > > >> > 2009-03-13 14:32:21,641 WARN > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: error getting > store > >> file > >> > index size for 310680591/cdr: java.io.FileNotFoundException: File does > >> not > >> > exist: > >> > > >> > hdfs://nd0-rack0-cloud:9000/hbase/TESTTAB/310680591/cdr/mapfiles/1049684492857034443/index > >> > at > >> > > >> > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:415) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HStoreFile.indexLength(HStoreFile.java:488) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HStore.getStorefilesIndexSize(HStore.java:2179) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.doMetrics(HRegionServer.java:941) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:334) > >> > at java.lang.Thread.run(Thread.java:619) > >> > > >> > 2009-03-13 14:32:21,641 ERROR > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Exceeded max > retries: > >> 10 > >> > java.io.FileNotFoundException: File does not exist: > >> > > >> > hdfs://nd0-rack0-cloud:9000/hbase/TESTTAB/310680591/cdr/mapfiles/1049684492857034443/index > >> > at > >> > > >> > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:415) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HStoreFile.indexLength(HStoreFile.java:488) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HStore.getStorefilesIndexSize(HStore.java:2179) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.createRegionLoad(HRegionServer.java:625) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:341) > >> > at java.lang.Thread.run(Thread.java:619) > >> > > >> > But it will not stop, still output this exception with "Exceeded max > >> > retries: 10". > >> > ....endless... > >> > > >> > ====================================== > >> > > >> > Before this exception, there is no other Exception on this node. > >> > But on other nodes: there are some following: > >> > > >> > 2009-03-13 09:22:19,389 ERROR > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: error closing and > >> > deleting HLog org.apache.hadoop.ipc.RemoteException: > java.io.IOException: > >> > Could not complete write to file > >> > /hbase/log_10.24.1.16_1236857785825_60020/hlog.dat.1236906952707 by > >> > DFSClient_-1142813574 at > >> > > >> > org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:378) > >> > at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) at > >> > > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > >> > at java.lang.reflect.Method.invoke(Method.java:597) at > >> > org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481) at > >> > org.apache.hadoop.ipc.Server$Handler.run(Server.java:894) at > >> > org.apache.hadoop.ipc.Client.call(Client.java:697) at > >> > org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216) at > >> > $Proxy1.complete(Unknown Source) at > >> > sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) at > >> > > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > >> > at java.lang.reflect.Method.invoke(Method.java:597) at > >> > > >> > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) > >> > at > >> > > >> > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) > >> > at $Proxy1.complete(Unknown Source) at > >> > > >> > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3130) > >> > at > >> > > >> > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3054) > >> > at > >> > > >> > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61) > >> > at > >> > org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86) > >> > at > org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:959) > >> at > >> > org.apache.hadoop.hbase.regionserver.HLog.close(HLog.java:421) at > >> > > org.apache.hadoop.hbase.regionserver.HLog.closeAndDelete(HLog.java:404) > >> at > >> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:373) > >> > at java.lang.Thread.run(Thread.java:619) > >> > > >> > > >> > and... > >> > > >> > 2009-03-13 10:03:40,149 ERROR > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner > >> > org.apache.hadoop.hbase.NotServingRegionException: > TESTTAB,,1236871823378 > >> at > >> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699) > >> > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) at > >> > > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > >> > at java.lang.reflect.Method.invoke(Method.java:597) at > >> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) at > >> > > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895) > >> > 2009-03-13 10:03:40,150 ERROR > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner > >> > org.apache.hadoop.hbase.NotServingRegionException: > TESTTAB,,1236871823378 > >> at > >> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699) > >> > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) at > >> > > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > >> > at java.lang.reflect.Method.invoke(Method.java:597) at > >> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) at > >> > > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895) > >> > 2009-03-13 10:03:40,150 ERROR > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner > >> > org.apache.hadoop.hbase.NotServingRegionException: > TESTTAB,,1236871823378 > >> at > >> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699) > >> > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) at > >> > > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > >> > at java.lang.reflect.Method.invoke(Method.java:597) at > >> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) at > >> > > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895) > >> > > >> > > >> > and ... > >> > > >> > 2009-03-13 13:01:58,220 ERROR > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner > >> > org.apache.hadoop.hbase.NotServingRegionException: > >> > CDRWAP,13576300...@2009-01-31 10:44:29.720,1236888415970 at > >> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699) > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > >> > > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > >> > at > >> > > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > >> > at java.lang.reflect.Method.invoke(Method.java:597) at > >> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) at > >> > > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895) > >> > 2009-03-13 13:01:58,348 INFO org.apache.hadoop.ipc.HBaseServer: IPC > >> Server > >> > handler 1 on 60020, call openScanner([...@7a5fe108, [...@2caf12fc, > >> [...@1a07754f, > >> > 9223372036854775807, > >> > org.apache.hadoop.hbase.filter.whilematchrowfil...@5fa6a2e2) from > >> > 10.24.1.10:44605: error: > >> org.apache.hadoop.hbase.NotServingRegionException: > >> > CDRWAP,13576300...@2009-01-31 10:44:29.720,1236888415970 > >> > org.apache.hadoop.hbase.NotServingRegionException: > >> > CDRWAP,13576300...@2009-01-31 10:44:29.720,1236888415970 at > >> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699) > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > >> > > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > >> > at > >> > > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > >> > at java.lang.reflect.Method.invoke(Method.java:597) at > >> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) at > >> > > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895) > >> > 2009-03-13 13:02:00,712 ERROR > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner > >> > org.apache.hadoop.hbase.NotServingRegionException: > >> > CDRWAP,13576300...@2009-01-31 10:44:29.720,1236888415970 at > >> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065) > >> > at > >> > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699) > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > >> > > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > >> > at > >> > > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > >> > at java.lang.reflect.Method.invoke(Method.java:597) at > >> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) at > >> > > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895) > >> > 2009-03-13 13:02:00,730 INFO org.apache.hadoop.ipc.HBaseServer: IPC > >> Server > >> > handler 8 on 60020, call openScanner([...@1eb57 > >> > > >> > > >
