Hi Jia, How is you HDFS running?
"Caused by: org.apache.hadoop.ipc. RemoteException(java.io.IOException): File /apps/hbase/data/lbc_zte_1_imei_index/4469e6b0500bf3f5ed0ac1247d249537/.tmp/e7bb489662344b26bc6de1e72c122eec could only be replicated to 0 nodes instead of minReplication (=1). There are 3 datanode(s) running and no node(s) are excluded in this operation. " Sound like there is some issues on the datanode. Have you checked its logs? JM 2013/8/13 jia.li <[email protected]> > > Hi , Devs/Users ; > Recently I use HBase API to insert big data into hbase;It's about 77G and > my cluster has one hbase-master,two hbase-regionservers ; > When the program executes a period of time, the regionserver automaticlly > shutdown. And I restart regionservers , > but this thing happens again . > from the regionserver log : > > 2013-08-13 12:12:08,983 FATAL > org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort: > loaded coprocessors are: > [org.apache.hadoop.hbase.coprocessor.AggregateImplementation, > com.zsmar.hbase.query.rowkey.RowKey3Endpoint] > 2013-08-13 12:12:08,988 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: > requestsPerSecond=49507, numberOfOnlineRegions=2260, numberOfStores=2260, > numberOfStorefiles=2368, storefileIndexSizeMB=12, rootIndexSizeKB=12786, > totalStaticIndexSizeKB=466828, totalStaticBloomSizeKB=54671, > memstoreSizeMB=711, mbInMemoryWithoutWAL=267, > numberOfPutsWithoutWAL=2409813, readRequestsCount=65095, > writeRequestsCount=299008, compactionQueueSize=92, flushQueueSize=0, > usedHeapMB=1182, maxHeapMB=1991, blockCacheSizeMB=54.35, > blockCacheFreeMB=443.57, blockCacheCount=5779, blockCacheHitCount=1455353, > blockCacheMissCount=240305, blockCacheEvictedCount=49, > blockCacheHitRatio=85%, blockCacheHitCachingRatio=99%, > hdfsBlocksLocalityIndex=75, slowHLogAppendCount=0, > fsReadLatencyHistogramMean=0, fsReadLatencyHistogramCount=0, > fsReadLatencyHistogramMedian=0, fsReadLatencyHistogram75th=0, > fsReadLatencyHistogram95th=0, fsReadLatencyHistogram99th=0, > fsReadLatencyHistogram999th=0, fsPreadLatencyHistogramMean=0, > fsPreadLatencyHistogramCount=0, fsPreadLatencyHistogramMedian=0, > fsPreadLatencyHistogram75th=0, fsPreadLatencyHistogram95th=0, > fsPreadLatencyHistogram99th=0, fsPreadLatencyHistogram999th=0, > fsWriteLatencyHistogramMean=0, fsWriteLatencyHistogramCount=0, > fsWriteLatencyHistogramMedian=0, fsWriteLatencyHistogram75th=0, > fsWriteLatencyHistogram95th=0, fsWriteLatencyHistogram99th=0, > fsWriteLatencyHistogram999th=0 > 2013-08-13 12:12:08,991 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Replay of HLog > required. Forcing server shutdown > 2013-08-13 12:12:08,991 INFO > org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Excluding unflushable > region > lbc_zte_1_nbr_index,436238C32A97DAB59D72E810C313CF4F100230,1376366743650.cf99f5047c85e155069c3970cdaf03c6. > - trying to find a different region to flush. > 2013-08-13 12:12:08,991 INFO > org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush of region > lbc_zte_1_imei_index,3333332A,1376364729049.4469e6b0500bf3f5ed0ac1247d249537. > due to global heap pressure > 2013-08-13 12:12:08,991 INFO org.apache.hadoop.ipc.HBaseServer: Stopping > server on 60020 > 2013-08-13 12:12:08,992 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC > Server handler 2 on 60020: exiting > 2013-08-13 12:12:08,992 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 8 on 60020: exiting > 2013-08-13 12:12:08,992 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC > Server handler 1 on 60020: exiting > 2013-08-13 12:12:08,992 INFO org.apache.hadoop.ipc.HBaseServer: Stopping > IPC Server listener on 60020 > 2013-08-13 12:12:08,992 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC > Server handler 6 on 60020: exiting > 2013-08-13 12:12:08,992 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC > Server handler 2 on 60020: exiting > 2013-08-13 12:12:08,992 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC > Server handler 0 on 60020: exiting > 2013-08-13 12:12:08,992 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 9 on 60020: exiting > 2013-08-13 12:12:08,992 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC > Server handler 1 on 60020: exiting > 2013-08-13 12:12:08,992 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 6 on 60020: exiting > 2013-08-13 12:12:08,992 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC > Server handler 9 on 60020: exiting > 2013-08-13 12:12:08,993 INFO > org.apache.hadoop.hbase.regionserver.SplitLogWorker: Sending interrupt to > stop the worker thread > 2013-08-13 12:12:08,993 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC > Server handler 8 on 60020: exiting > 2013-08-13 12:12:08,992 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC > Server handler 3 on 60020: exiting > 2013-08-13 12:12:08,992 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 7 on 60020: exiting > 2013-08-13 12:12:08,992 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 2 on 60020: exiting > 2013-08-13 12:12:08,993 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC > Server handler 5 on 60020: exiting > 2013-08-13 12:12:08,993 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC > Server handler 0 on 60020: exiting > 2013-08-13 12:12:08,993 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 5 on 60020: exiting > 2013-08-13 12:12:08,993 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 3 on 60020: exiting > 2013-08-13 12:12:08,993 INFO org.apache.hadoop.ipc.HBaseServer: Stopping > IPC Server Responder > 2013-08-13 12:12:08,994 INFO org.apache.hadoop.ipc.HBaseServer: Stopping > IPC Server Responder > 2013-08-13 12:12:08,993 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 0 on 60020: exiting > 2013-08-13 12:12:08,993 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer > 2013-08-13 12:12:08,993 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC > Server handler 7 on 60020: exiting > 2013-08-13 12:12:08,993 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 1 on 60020: exiting > 2013-08-13 12:12:08,993 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 4 on 60020: exiting > 2013-08-13 12:12:08,993 INFO > org.apache.hadoop.hbase.regionserver.SplitLogWorker: SplitLogWorker > interrupted while waiting for task, exiting: java.lang.InterruptedException > 2013-08-13 12:12:08,998 INFO > org.apache.hadoop.hbase.regionserver.SplitLogWorker: SplitLogWorker > phd03.hadoop.audaque.com,60020,1376364509049 exiting > 2013-08-13 12:12:09,023 INFO org.mortbay.log: Stopped > [email protected]:60030 > 2013-08-13 12:12:09,029 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server > Responder, call delete([B@69fba6f8, > {"ts":9223372036854775807,"totalColumns":2,"families":{"info":[{"timestamp":1376367128966,"qualifier":"splitA","vlen":0},{"timestamp":1376367128966,"qualifier":"splitB","vlen":0}]},"row":"lbc_zte_1,063AE3C37783FD39EE2142BEE43576C2200901,1376366676823.9b4e8b4ce35bf541fc1e9b5b77a22b62."}), > rpc version=1, client version=29, methodsFingerPrint=-56040613 from > 172.16.1.91:46113: output error > 2013-08-13 12:12:09,032 WARN org.apache.hadoop.ipc.HBaseServer: PRI IPC > Server handler 4 on 60020 caught a ClosedChannelException, this means that > the server was processing a request but the client went away. The error > message was: null > 2013-08-13 12:12:09,032 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC > Server handler 4 on 60020: exiting > 2013-08-13 12:12:09,038 WARN > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > Failed all from region=.META.,,1.1028785192, hostname= > phd03.hadoop.audaque.com, port=60020 > java.util.concurrent.ExecutionException: java.io.IOException: Call to > phd03.hadoop.audaque.com/172.16.1.93:60020 failed on local exception: > java.io.EOFException > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) > at java.util.concurrent.FutureTask.get(FutureTask.java:83) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1544) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1396) > at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:918) > at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:774) > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:749) > at org.apache.hadoop.hbase.catalog.MetaEditor.put(MetaEditor.java:99) > at > org.apache.hadoop.hbase.catalog.MetaEditor.putToMetaTable(MetaEditor.java:66) > at > org.apache.hadoop.hbase.catalog.MetaEditor.offlineParentInMeta(MetaEditor.java:188) > at > org.apache.hadoop.hbase.regionserver.SplitTransaction.createDaughters(SplitTransaction.java:327) > at > org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:457) > at > org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:67) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.io.IOException: Call to > phd03.hadoop.audaque.com/172.16.1.93:60020 failed on local exception: > java.io.EOFException > at > org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:1056) > at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1025) > at > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150) > at com.sun.proxy.$Proxy20.multi(Unknown Source) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1373) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1371) > at > org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.java:210) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1380) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1368) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > ... 3 more > Caused by: java.io.EOFException > at java.io.DataInputStream.readInt(DataInputStream.java:375) > at > org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:672) > at > org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:606) > 2013-08-13 12:12:09,131 INFO > org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for > hdfs://phd01:8020/apps/hbase/data/lbc_zte_1_imei_index/4469e6b0500bf3f5ed0ac1247d249537/.tmp/e7bb489662344b26bc6de1e72c122eec: > ROW, CompoundBloomFilterWriter > 2013-08-13 12:12:09,131 INFO > org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter > type for > hdfs://phd01:8020/apps/hbase/data/lbc_zte_1_imei_index/4469e6b0500bf3f5ed0ac1247d249537/.tmp/e7bb489662344b26bc6de1e72c122eec: > CompoundBloomFilterWriter > 2013-08-13 12:12:09,138 WARN org.apache.hadoop.hdfs.DFSClient: > DataStreamer Exception > org.apache.hadoop.ipc.RemoteException(java.io.IOException): File > /apps/hbase/data/lbc_zte_1_imei_index/4469e6b0500bf3f5ed0ac1247d249537/.tmp/e7bb489662344b26bc6de1e72c122eec > could only be replicated to 0 nodes instead of minReplication (=1). There > are 3 datanode(s) running and no node(s) are excluded in this operation. > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1322) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2369) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:469) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:300) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:45843) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1694) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1690) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1688) > at org.apache.hadoop.ipc.Client.call(Client.java:1164) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202) > at com.sun.proxy.$Proxy14.addBlock(Unknown Source) > at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83) > at com.sun.proxy.$Proxy14.addBlock(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:288) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1150) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1003) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:463) > 2013-08-13 12:12:09,140 FATAL > org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server > phd03.hadoop.audaque.com,60020,1376364509049: Replay of HLog required. > Forcing server shutdown > org.apache.hadoop.hbase.DroppedSnapshotException: region: > lbc_zte_1_imei_index,3333332A,1376364729049.4469e6b0500bf3f5ed0ac1247d249537. > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1472) > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1351) > at > org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1292) > at > org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:406) > at > org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:202) > at > org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223) > at java.lang.Thread.run(Thread.java:662) > Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): > File > /apps/hbase/data/lbc_zte_1_imei_index/4469e6b0500bf3f5ed0ac1247d249537/.tmp/e7bb489662344b26bc6de1e72c122eec > could only be replicated to 0 nodes instead of minReplication (=1). There > are 3 datanode(s) running and no node(s) are excluded in this operation. > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1322) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2369) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:469) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:300) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:45843) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1694) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1690) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1688) > at org.apache.hadoop.ipc.Client.call(Client.java:1164) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202) > at com.sun.proxy.$Proxy14.addBlock(Unknown Source) > at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83) > at com.sun.proxy.$Proxy14.addBlock(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:288) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1150) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1003) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:463) > 2013-08-13 12:12:09,140 FATAL > org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort: > loaded coprocessors are: > [org.apache.hadoop.hbase.coprocessor.AggregateImplementation, > com.zsmar.hbase.query.rowkey.RowKey3Endpoint] > 2013-08-13 12:12:09,140 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: > requestsPerSecond=49507, numberOfOnlineRegions=2260, numberOfStores=2260, > numberOfStorefiles=2368, storefileIndexSizeMB=12, rootIndexSizeKB=12786, > totalStaticIndexSizeKB=466828, totalStaticBloomSizeKB=54671, > memstoreSizeMB=711, mbInMemoryWithoutWAL=267, > numberOfPutsWithoutWAL=2409813, readRequestsCount=65095, > writeRequestsCount=299008, compactionQueueSize=92, flushQueueSize=0, > usedHeapMB=1173, maxHeapMB=1991, blockCacheSizeMB=54.35, > blockCacheFreeMB=443.57, blockCacheCount=5779, blockCacheHitCount=1455353, > blockCacheMissCount=240305, blockCacheEvictedCount=49, > blockCacheHitRatio=85%, blockCacheHitCachingRatio=99%, > hdfsBlocksLocalityIndex=75, slowHLogAppendCount=0, > fsReadLatencyHistogramMean=0, fsReadLatencyHistogramCount=0, > fsReadLatencyHistogramMedian=0, fsReadLatencyHistogram75th=0, > fsReadLatencyHistogram95th=0, fsReadLatencyHistogram99th=0, > fsReadLatencyHistogram999th=0, fsPreadLatencyHistogramMean=0, > fsPreadLatencyHistogramCount=0, fsPreadLatencyHistogramMedian=0, > fsPreadLatencyHistogram75th=0, fsPreadLatencyHistogram95th=0, > fsPreadLatencyHistogram99th=0, fsPreadLatencyHistogram999th=0, > fsWriteLatencyHistogramMean=0, fsWriteLatencyHistogramCount=0, > fsWriteLatencyHistogramMedian=0, fsWriteLatencyHistogram75th=0, > fsWriteLatencyHistogram95th=0, fsWriteLatencyHistogram99th=0, > fsWriteLatencyHistogram999th=0 > 2013-08-13 12:12:09,145 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Replay of HLog > required. Forcing server shutdown > 2013-08-13 12:12:09,146 INFO > org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting. > > > could someone know the reason why it happens ? and give some mesages . > > 2013-08-13 > > > > 李 佳(套马的汉子，你威武雄壮) > 华傲数据技术有限公司 > 深圳市南山区高新区中区高新中一道9号软件大厦713室 > Mobile : 18665837392 > Email: [email protected] >
