hi haosdent, lin shuai the root caused is found. this is blocked by vmtool. this is vsphere's application.
2016-02-04 11:03 GMT+08:00 haosdent <[email protected]>: > How about the "/health" entrypoint in slave? > > On Thu, Feb 4, 2016 at 10:57 AM, tommy xiao <[email protected]> wrote: > >> *Shuai Lin，* >> >> *but the /help entrypoint is ok in docker/vm.this only affect state.json >> entrypoint.* >> >> 2016-02-04 9:06 GMT+08:00 Shuai Lin <[email protected]>: >> >>> I would suggest first check the possibility of whether it's a problem >>> of the vm/docker networking, e.g. run a web server in docker in the vm, and >>> try to download some files from it, and vice versa. >>> >>> On Thu, Feb 4, 2016 at 2:17 AM, tommy xiao <[email protected]> wrote: >>> >>>> hi, All >>>> >>>> I came across another case. the user is use vmware vsphere to setup a >>>> vm. then start three ubuntu servers. install mesos-slave. the mesos-slave >>>> state.json always take about some minutes to result json. i have already >>>> check the env, we can't found any different settings. >>>> >>>> because the mesos-slave is in docker, when i use docker restart >>>> mesos-slave. the http://xxx.xx.xx:5051/state.json will quickly >>>> response. this is reproduce case in mesos 0.23 version. but i testing this >>>> env in my local vsphere, the cluster can setup correctly. it let me very >>>> confuse. anyone can give some advise on it? >>>> >>>> >>>> >>>> >>>> 2016-02-03 15:56 GMT+08:00 tommy xiao <[email protected]>: >>>> >>>>> the final result, this is caused by ksoftirqd eats 100% of one CPU >>>>> core. there is not mesos-slave's issue, even i installed the optimized >>>>> binary. >>>>> >>>>> 2016-02-03 12:13 GMT+08:00 tommy xiao <[email protected]>: >>>>> >>>>>> today, i have testing the issued cluster, found the ksoftirqd eats >>>>>> 100% of one CPU core, then the mesos-slave state.json will get timeout. >>>>>> i >>>>>> will later testing the O2 optimized mesos-slave binary again. then i will >>>>>> report the result later. >>>>>> >>>>>> 2016-02-02 23:41 GMT+08:00 tommy xiao <[email protected]>: >>>>>> >>>>>>> haosdent, tomorrow i will testing the rebuild package on the exists >>>>>>> cluster. please hold on. >>>>>>> >>>>>>> 2016-02-02 23:27 GMT+08:00 haosdent <[email protected]>: >>>>>>> >>>>>>>> does the performance problem still exists? >>>>>>>> On Feb 2, 2016 11:17 PM, "tommy xiao" <[email protected]> wrote: >>>>>>>> >>>>>>>>> with test result, when i remove CFLAGS='-g -O2 -w' . it works now. >>>>>>>>> >>>>>>>>> make -j2 CXXFLAGS='-g -O2 -w -std=c++11' >>>>>>>>> >>>>>>>>> 2016-02-02 22:38 GMT+08:00 tommy xiao <[email protected]>: >>>>>>>>> >>>>>>>>>> just make sure the libleveldb-dev is not relative with mesos >>>>>>>>>> build. even without libleveldb-dev, i also can't build successful. >>>>>>>>>> if i >>>>>>>>>> remove O2, just default O0, the make build will get successful. it >>>>>>>>>> is weird. >>>>>>>>>> >>>>>>>>>> 2016-02-02 19:24 GMT+08:00 haosdent <[email protected]>: >>>>>>>>>> >>>>>>>>>>> Why you execute `sudo apt-get install libleveldb-dev` before >>>>>>>>>>> compile? Mesos have a bundle leveldb package. >>>>>>>>>>> >>>>>>>>>>> On Tue, Feb 2, 2016 at 7:21 PM, tommy xiao <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> David, >>>>>>>>>>>> >>>>>>>>>>>> i am follow your suggest, and build with O2, came cross leveldb >>>>>>>>>>>> error. >>>>>>>>>>>> this is ubuntu server, i use below instruction: >>>>>>>>>>>> sudo apt-get install libleveldb-dev >>>>>>>>>>>> git clone https://git-wip-us.apache.org/repos/asf/mesos.git >>>>>>>>>>>> cd mesos >>>>>>>>>>>> ./bootstrap >>>>>>>>>>>> mkdir build && cd build && ../configure >>>>>>>>>>>> make -j2 CFLAGS='-g -O2 -w' CXXFLAGS='-g -O2 -w -std=c++11' >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> cd leveldb && \ >>>>>>>>>>>> >>>>>>>>>>>> make CC="gcc" CXX="g++" OPT="-g -O2 -w -std=c++11 >>>>>>>>>>>> -fPIC" >>>>>>>>>>>> >>>>>>>>>>>> make[4]: Entering directory >>>>>>>>>>>> `/home/dsxiao/mesos/build/3rdparty/leveldb' >>>>>>>>>>>> >>>>>>>>>>>> g++ -pthread -shared -Wl,-soname >>>>>>>>>>>> -Wl,/home/dsxiao/mesos/build/3rdparty/leveldb/libleveldb.so.1 -g >>>>>>>>>>>> -O2 -w >>>>>>>>>>>> -std=c++11 -fPIC db/builder.cc db/c.cc db/dbformat.cc db/db_impl.cc >>>>>>>>>>>> db/db_iter.cc db/filename.cc db/log_reader.cc db/log_writer.cc >>>>>>>>>>>> db/memtable.cc db/repair.cc db/table_cache.cc db/version_edit.cc >>>>>>>>>>>> db/version_set.cc db/write_batch.cc table/block_builder.cc >>>>>>>>>>>> table/block.cc >>>>>>>>>>>> table/filter_block.cc table/format.cc table/iterator.cc >>>>>>>>>>>> table/merger.cc >>>>>>>>>>>> table/table_builder.cc table/table.cc table/two_level_iterator.cc >>>>>>>>>>>> util/arena.cc util/bloom.cc util/cache.cc util/coding.cc >>>>>>>>>>>> util/comparator.cc >>>>>>>>>>>> util/crc32c.cc util/env.cc util/env_posix.cc util/filter_policy.cc >>>>>>>>>>>> util/hash.cc util/histogram.cc util/logging.cc util/options.cc >>>>>>>>>>>> util/status.cc port/port_posix.cc -o libleveldb.so.1.4 >>>>>>>>>>>> >>>>>>>>>>>> g++ -g -O2 -w -std=c++11 -c db/builder.cc -o db/builder.o >>>>>>>>>>>> >>>>>>>>>>>> db/builder.cc:5:24: fatal error: db/builder.h: No such file or >>>>>>>>>>>> directory >>>>>>>>>>>> >>>>>>>>>>>> #include "db/builder.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> db/builder.cc:5:24: fatal error: db/builder.h: No such file or >>>>>>>>>>>> directory >>>>>>>>>>>> >>>>>>>>>>>> #include "db/builder.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> make[4]: *** [db/builder.o] Error 1 >>>>>>>>>>>> >>>>>>>>>>>> make[4]: *** Waiting for unfinished jobs.... >>>>>>>>>>>> >>>>>>>>>>>> db/dbformat.cc:6:25: fatal error: db/dbformat.h: No such file >>>>>>>>>>>> or directory >>>>>>>>>>>> >>>>>>>>>>>> #include "db/dbformat.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> db/db_impl.cc:5:24: fatal error: db/db_impl.h: No such file or >>>>>>>>>>>> directory >>>>>>>>>>>> >>>>>>>>>>>> #include "db/db_impl.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> db/db_iter.cc:5:24: fatal error: db/db_iter.h: No such file or >>>>>>>>>>>> directory >>>>>>>>>>>> >>>>>>>>>>>> #include "db/db_iter.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> db/filename.cc:7:25: fatal error: db/filename.h: No such file >>>>>>>>>>>> or directory >>>>>>>>>>>> >>>>>>>>>>>> #include "db/filename.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> db/log_reader.cc:5:27: fatal error: db/log_reader.h: No such >>>>>>>>>>>> file or directory >>>>>>>>>>>> >>>>>>>>>>>> #include "db/log_reader.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> db/log_writer.cc:5:27: fatal error: db/log_writer.h: No such >>>>>>>>>>>> file or directory >>>>>>>>>>>> >>>>>>>>>>>> #include "db/log_writer.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> db/memtable.cc:5:25: fatal error: db/memtable.h: No such file >>>>>>>>>>>> or directory >>>>>>>>>>>> >>>>>>>>>>>> #include "db/memtable.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> db/repair.cc:27:24: fatal error: db/builder.h: No such file or >>>>>>>>>>>> directory >>>>>>>>>>>> >>>>>>>>>>>> #include "db/builder.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> db/table_cache.cc:5:28: fatal error: db/table_cache.h: No such >>>>>>>>>>>> file or directory >>>>>>>>>>>> >>>>>>>>>>>> #include "db/table_cache.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> db/version_edit.cc:5:29: fatal error: db/version_edit.h: No >>>>>>>>>>>> such file or directory >>>>>>>>>>>> >>>>>>>>>>>> #include "db/version_edit.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> db/version_set.cc:5:28: fatal error: db/version_set.h: No such >>>>>>>>>>>> file or directory >>>>>>>>>>>> >>>>>>>>>>>> #include "db/version_set.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> db/write_batch.cc:19:25: fatal error: db/dbformat.h: No such >>>>>>>>>>>> file or directory >>>>>>>>>>>> >>>>>>>>>>>> #include "db/dbformat.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> table/block_builder.cc:29:33: fatal error: >>>>>>>>>>>> table/block_builder.h: No such file or directory >>>>>>>>>>>> >>>>>>>>>>>> #include "table/block_builder.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> table/block.cc:7:25: fatal error: table/block.h: No such file >>>>>>>>>>>> or directory >>>>>>>>>>>> >>>>>>>>>>>> #include "table/block.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> table/filter_block.cc:5:32: fatal error: table/filter_block.h: >>>>>>>>>>>> No such file or directory >>>>>>>>>>>> >>>>>>>>>>>> #include "table/filter_block.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> table/format.cc:5:26: fatal error: table/format.h: No such file >>>>>>>>>>>> or directory >>>>>>>>>>>> >>>>>>>>>>>> #include "table/format.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> table/merger.cc:5:26: fatal error: table/merger.h: No such file >>>>>>>>>>>> or directory >>>>>>>>>>>> >>>>>>>>>>>> #include "table/merger.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> table/table_builder.cc:12:33: fatal error: >>>>>>>>>>>> table/block_builder.h: No such file or directory >>>>>>>>>>>> >>>>>>>>>>>> #include "table/block_builder.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> table/table.cc:12:25: fatal error: table/block.h: No such file >>>>>>>>>>>> or directory >>>>>>>>>>>> >>>>>>>>>>>> #include "table/block.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> table/two_level_iterator.cc:5:38: fatal error: >>>>>>>>>>>> table/two_level_iterator.h: No such file or directory >>>>>>>>>>>> >>>>>>>>>>>> #include "table/two_level_iterator.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> util/arena.cc:5:24: fatal error: util/arena.h: No such file or >>>>>>>>>>>> directory >>>>>>>>>>>> >>>>>>>>>>>> #include "util/arena.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> util/bloom.cc:8:23: fatal error: util/hash.h: No such file or >>>>>>>>>>>> directory >>>>>>>>>>>> >>>>>>>>>>>> #include "util/hash.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> util/cache.cc:10:23: fatal error: port/port.h: No such file or >>>>>>>>>>>> directory >>>>>>>>>>>> >>>>>>>>>>>> #include "port/port.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> util/coding.cc:5:25: fatal error: util/coding.h: No such file >>>>>>>>>>>> or directory >>>>>>>>>>>> >>>>>>>>>>>> #include "util/coding.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> util/comparator.cc:9:26: fatal error: util/logging.h: No such >>>>>>>>>>>> file or directory >>>>>>>>>>>> >>>>>>>>>>>> #include "util/logging.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> util/crc32c.cc:8:25: fatal error: util/crc32c.h: No such file >>>>>>>>>>>> or directory >>>>>>>>>>>> >>>>>>>>>>>> #include "util/crc32c.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> util/env_posix.cc:24:23: fatal error: port/port.h: No such file >>>>>>>>>>>> or directory >>>>>>>>>>>> >>>>>>>>>>>> #include "port/port.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> util/hash.cc:6:25: fatal error: util/coding.h: No such file or >>>>>>>>>>>> directory >>>>>>>>>>>> >>>>>>>>>>>> #include "util/coding.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> util/histogram.cc:7:23: fatal error: port/port.h: No such file >>>>>>>>>>>> or directory >>>>>>>>>>>> >>>>>>>>>>>> #include "port/port.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> util/logging.cc:5:26: fatal error: util/logging.h: No such file >>>>>>>>>>>> or directory >>>>>>>>>>>> >>>>>>>>>>>> #include "util/logging.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> util/status.cc:6:23: fatal error: port/port.h: No such file or >>>>>>>>>>>> directory >>>>>>>>>>>> >>>>>>>>>>>> #include "port/port.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> port/port_posix.cc:5:29: fatal error: port/port_posix.h: No >>>>>>>>>>>> such file or directory >>>>>>>>>>>> >>>>>>>>>>>> #include "port/port_posix.h" >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> compilation terminated. >>>>>>>>>>>> >>>>>>>>>>>> make[4]: *** [libleveldb.so.1.4] Error 1 >>>>>>>>>>>> >>>>>>>>>>>> make[4]: Leaving directory >>>>>>>>>>>> `/home/dsxiao/mesos/build/3rdparty/leveldb' >>>>>>>>>>>> >>>>>>>>>>>> make[3]: *** [leveldb/libleveldb.a] Error 2 >>>>>>>>>>>> >>>>>>>>>>>> make[3]: Leaving directory `/home/dsxiao/mesos/build/3rdparty' >>>>>>>>>>>> >>>>>>>>>>>> make[2]: *** [all-recursive] Error 1 >>>>>>>>>>>> >>>>>>>>>>>> make[2]: Leaving directory `/home/dsxiao/mesos/build/3rdparty' >>>>>>>>>>>> >>>>>>>>>>>> make[1]: *** [all] Error 2 >>>>>>>>>>>> >>>>>>>>>>>> make[1]: Leaving directory `/home/dsxiao/mesos/build/3rdparty' >>>>>>>>>>>> >>>>>>>>>>>> make: *** [all-recursive] Error 1 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 2016-02-01 20:09 GMT+08:00 David J. Palaitis < >>>>>>>>>>>> [email protected]>: >>>>>>>>>>>> >>>>>>>>>>>>> We've come across this issue in 0.23. We rebuilt 0.23 with >>>>>>>>>>>>> full optimization, O2, and the issue was resolved. Try it out and >>>>>>>>>>>>> let me >>>>>>>>>>>>> know how it goes. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Feb 1, 2016 at 7:02 AM, tommy xiao <[email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> In bare mental server, the 5051 port, the state.json query >>>>>>>>>>>>>> will hang a long time to response the json data. it about 5 >>>>>>>>>>>>>> minutes. the >>>>>>>>>>>>>> curious thing is the 5051 port, the /help command can work >>>>>>>>>>>>>> correctly. so i >>>>>>>>>>>>>> wonder to know, anyone came across the case in any time? no >>>>>>>>>>>>>> clues to debug >>>>>>>>>>>>>> in my view. >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Deshi Xiao >>>>>>>>>>>>>> Twitter: xds2000 >>>>>>>>>>>>>> E-mail: xiaods(AT)gmail.com >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Deshi Xiao >>>>>>>>>>>> Twitter: xds2000 >>>>>>>>>>>> E-mail: xiaods(AT)gmail.com >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Best Regards, >>>>>>>>>>> Haosdent Huang >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Deshi Xiao >>>>>>>>>> Twitter: xds2000 >>>>>>>>>> E-mail: xiaods(AT)gmail.com >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Deshi Xiao >>>>>>>>> Twitter: xds2000 >>>>>>>>> E-mail: xiaods(AT)gmail.com >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Deshi Xiao >>>>>>> Twitter: xds2000 >>>>>>> E-mail: xiaods(AT)gmail.com >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Deshi Xiao >>>>>> Twitter: xds2000 >>>>>> E-mail: xiaods(AT)gmail.com >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Deshi Xiao >>>>> Twitter: xds2000 >>>>> E-mail: xiaods(AT)gmail.com >>>>> >>>> >>>> >>>> >>>> -- >>>> Deshi Xiao >>>> Twitter: xds2000 >>>> E-mail: xiaods(AT)gmail.com >>>> >>> >>> >> >> >> -- >> Deshi Xiao >> Twitter: xds2000 >> E-mail: xiaods(AT)gmail.com >> > > > > -- > Best Regards, > Haosdent Huang > -- Deshi Xiao Twitter: xds2000 E-mail: xiaods(AT)gmail.com
