Hi, All: I'm doing some work on hive to make it support multi-distinct clause in one statement these month. I almost archieve my goal except for some failures on unit test. And I believe in that these failure were caused by system environment.
Hive version was hive realease 0.4.0 rc2. My colleagues and I have did some work on this version, such as adding ACL layer in hive. Failure appears while running unit test TestCliDriver and the failed .q file were input41.q and input_part9.q. The expected result of query "" is 500,0 while the actual result was 0, 500 on one node called "hdp2". But on another machine (called dw81), unit test passed with the same code. There are three stages for this query. Frist two stages gets result of "select count(1) as cnt from src" and "select count(1) as cnt from srcpart where ds = '2009-08-09'", which are 500 and 0, and stores them as files. The third stage reads two files and joins them to get final result " 500 0". I compared the difference between the two node and found that this two node will get different results while listing directory. The listing program were written in C and Java as attachments. On "hdp2", results are $ ./a.out test filename:.. d_info:13700925 d_reclen:24s filename:. d_info:13977094 d_reclen:24s filename:file1 d_info:13979509 d_reclen:32s filename:file0 d_info:13977965 d_reclen:32s and $ java -cp . ls file1 file0 , while one "dw81", results are $ ./a.out test filename:.. d_info:79729027 d_reclen:24s filename:file0 d_info:79729540 d_reclen:32s filename:. d_info:79729531 d_reclen:24s filename:file1 d_info:79729536 d_reclen:32s and $ java -cp . ls file0 file1 There is also listing operation on stage three and the result of stage three depends on the order of file listing. I traced the output files, name of output file of stage-1 is "build/ql/tmp/485971631/10001/attempt_local_0001_m_000000_0" with value of 500 and corresponding name and value of stage-2 are "build/ql/tmp/485971631/10001/attempt_local_0001_m_000001_0" and 0. On "hdp2", when I list directory "build/ql/tmp/485971631/10001/", I'll get build/ql/tmp/485971631/10001/attempt_local_0001_m_000001_0 build/ql/tmp/485971631/10001/attempt_local_0001_m_000000_0 and values are 0 500 It was wrong output. While on "dw81", I'll get build/ql/tmp/485971631/10001/attempt_local_0001_m_000000_0 build/ql/tmp/485971631/10001/attempt_local_0001_m_000001_0 and values are 500 0 which is the expected result. So I believe that the output results of the operation depend on the order of file-listing of certain directory, which may be uncontrollable on different platform. Regards. Mafish
