Hi All,
I see very strange picture when running hawq TPC-DS benchmark. The data generation phase for 500BG data set showed 1.9GB/sec through put o= n our 9 node Hadoop cluster. The table analyze phase showed 3.2GB/sec throughput. However the test itsel= f shows very poor HDFS performance: * test run as: ./rollout.sh 100 false tpcds true 5 true true true true = true true true true true 1 * ~90MB/sec for read and writes clusterwide. I've seen 1.9GB/sec during= dataload and table analyze phase. * 72 postgres processes on each datanode and they consume 80% - 90% of = CPU each and 0% MEM, doing very little I/O. * 35036 files on HDFS with 2MB size per each file. Is this normal? Our = block size is 128MB * Top example on one of the nodes shows 0% memory allocated for Postgre= s but processes are heavily busy: Cpu(s): 0.0%us, 77.4%sy, 0.9%ni, 21.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0= %st Mem: 264403536k total, 172780840k used, 91622696k free, 2986328k buffers Swap: 4194300k total, 0k used, 4194300k free, 155959360k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 486160 gpadmin 39 19 872m 28m 10m D 88.7 0.0 4:32.52 postgres 486926 gpadmin 39 19 872m 28m 10m R 87.4 0.0 4:34.75 postgres 486405 gpadmin 39 19 872m 28m 10m R 86.4 0.0 4:23.99 postgres 487162 gpadmin 39 19 872m 28m 10m R 80.4 0.0 4:30.14 postgres 486761 gpadmin 39 19 872m 28m 10m R 78.8 0.0 4:28.41 postgres 486256 gpadmin 39 19 872m 28m 10m D 76.5 0.0 4:30.63 postgres Please suggest explanations why this happens. ________________________________ This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), its subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.
