gabotechs commented on issue #19784: URL: https://github.com/apache/datafusion/issues/19784#issuecomment-3755083882
I'm seeing something weird in TPC-DS Q14, something that the tests in https://github.com/datafusion-contrib/datafusion-distributed/ seem to be catching, originally reported here https://github.com/pydantic/datafusion-distributed/pull/2#discussion_r2693942638. It seems like the returned results for TPC-DS Q14 are different from version 51 to 52: <details><summary>Result of pretty_format_batches for TPC-DS Q14 in v51</summary> ``` +---------+------------+------------+---------------+--------------------+------------------+ | channel | i_brand_id | i_class_id | i_category_id | sum_sales | sum_number_sales | +---------+------------+------------+---------------+--------------------+------------------+ | | | | | 674173362.5100003 | 155629 | | catalog | | | | 237410857.47000003 | 46322 | | catalog | 1001001.0 | | | 1697729.02 | 347 | | catalog | 1001001.0 | 1.0 | | 855204.24 | 167 | | catalog | 1001001.0 | 1.0 | 1.0 | 115019.61000000002 | 20 | | catalog | 1001001.0 | 1.0 | 2.0 | 146344.46999999997 | 27 | | catalog | 1001001.0 | 1.0 | 3.0 | 22597.190000000002 | 3 | | catalog | 1001001.0 | 1.0 | 4.0 | 107555.42999999998 | 23 | | catalog | 1001001.0 | 1.0 | 5.0 | 122521.31000000001 | 25 | | catalog | 1001001.0 | 1.0 | 6.0 | 16883.97 | 3 | | catalog | 1001001.0 | 1.0 | 7.0 | 46329.78 | 9 | | catalog | 1001001.0 | 1.0 | 8.0 | 77861.85 | 13 | | catalog | 1001001.0 | 1.0 | 9.0 | 99985.34999999999 | 21 | | catalog | 1001001.0 | 1.0 | 10.0 | 100105.28000000001 | 23 | | catalog | 1001001.0 | 2.0 | | 125167.21999999997 | 24 | | catalog | 1001001.0 | 2.0 | 2.0 | 43967.969999999994 | 7 | | catalog | 1001001.0 | 2.0 | 3.0 | 68565.37999999999 | 14 | | catalog | 1001001.0 | 2.0 | 5.0 | 12633.87 | 3 | | catalog | 1001001.0 | 3.0 | | 198685.08000000002 | 43 | | catalog | 1001001.0 | 3.0 | 1.0 | 11100.79 | 5 | | catalog | 1001001.0 | 3.0 | 2.0 | 60551.64 | 14 | | catalog | 1001001.0 | 3.0 | 4.0 | 28455.230000000003 | 4 | | catalog | 1001001.0 | 3.0 | 6.0 | 36821.61 | 7 | | catalog | 1001001.0 | 3.0 | 7.0 | 17250.82 | 6 | | catalog | 1001001.0 | 3.0 | 8.0 | 14426.920000000002 | 4 | | catalog | 1001001.0 | 3.0 | 9.0 | 30078.07 | 3 | | catalog | 1001001.0 | 4.0 | | 109585.96999999999 | 31 | | catalog | 1001001.0 | 4.0 | 2.0 | 45473.85 | 13 | | catalog | 1001001.0 | 4.0 | 3.0 | 16558.92 | 8 | | catalog | 1001001.0 | 4.0 | 4.0 | 47553.2 | 10 | | catalog | 1001001.0 | 5.0 | | 59790.61 | 17 | | catalog | 1001001.0 | 5.0 | 9.0 | 30112.109999999997 | 12 | | catalog | 1001001.0 | 5.0 | 10.0 | 29678.5 | 5 | | catalog | 1001001.0 | 6.0 | | 10261.82 | 3 | | catalog | 1001001.0 | 6.0 | 9.0 | 10261.82 | 3 | | catalog | 1001001.0 | 7.0 | | 18244.94 | 3 | | catalog | 1001001.0 | 7.0 | 7.0 | 18244.94 | 3 | | catalog | 1001001.0 | 8.0 | | 55768.46 | 13 | | catalog | 1001001.0 | 8.0 | 7.0 | 28872.49 | 7 | | catalog | 1001001.0 | 8.0 | 10.0 | 26895.969999999998 | 6 | | catalog | 1001001.0 | 9.0 | | 30944.190000000002 | 5 | | catalog | 1001001.0 | 9.0 | 6.0 | 30944.190000000002 | 5 | | catalog | 1001001.0 | 11.0 | | 82810.87 | 12 | | catalog | 1001001.0 | 11.0 | 9.0 | 82810.87 | 12 | | catalog | 1001001.0 | 12.0 | | 38427.520000000004 | 9 | | catalog | 1001001.0 | 12.0 | 10.0 | 38427.520000000004 | 9 | | catalog | 1001001.0 | 15.0 | | 112838.1 | 20 | | catalog | 1001001.0 | 15.0 | 9.0 | 53508.79000000001 | 7 | | catalog | 1001001.0 | 15.0 | 10.0 | 59329.31 | 13 | | catalog | 1001002.0 | | | 3527831.3300000005 | 706 | | catalog | 1001002.0 | 1.0 | | 2673969.8900000006 | 530 | | catalog | 1001002.0 | 1.0 | 1.0 | 2673969.8900000006 | 530 | | catalog | 1001002.0 | 2.0 | | 140831.91000000003 | 29 | | catalog | 1001002.0 | 2.0 | 1.0 | 140831.91000000003 | 29 | | catalog | 1001002.0 | 3.0 | | 320175.86999999994 | 67 | | catalog | 1001002.0 | 3.0 | 1.0 | 320175.86999999994 | 67 | | catalog | 1001002.0 | 4.0 | | 133287.96000000002 | 21 | | catalog | 1001002.0 | 4.0 | 1.0 | 133287.96000000002 | 21 | | catalog | 1001002.0 | 5.0 | | 16606.9 | 9 | | catalog | 1001002.0 | 5.0 | 1.0 | 16606.9 | 9 | | catalog | 1001002.0 | 6.0 | | 15133.01 | 4 | | catalog | 1001002.0 | 6.0 | 1.0 | 15133.01 | 4 | | catalog | 1001002.0 | 7.0 | | 24471.260000000002 | 10 | | catalog | 1001002.0 | 7.0 | 1.0 | 24471.260000000002 | 10 | | catalog | 1001002.0 | 8.0 | | 63773.049999999996 | 12 | | catalog | 1001002.0 | 8.0 | 1.0 | 63773.049999999996 | 12 | | catalog | 1001002.0 | 9.0 | | 9167.189999999999 | 3 | | catalog | 1001002.0 | 9.0 | 1.0 | 9167.189999999999 | 3 | | catalog | 1001002.0 | 12.0 | | 29108.420000000002 | 7 | | catalog | 1001002.0 | 12.0 | 1.0 | 29108.420000000002 | 7 | | catalog | 1001002.0 | 15.0 | | 31143.449999999997 | 6 | | catalog | 1001002.0 | 15.0 | 1.0 | 31143.449999999997 | 6 | | catalog | 1001002.0 | 16.0 | | 70162.42 | 8 | | catalog | 1001002.0 | 16.0 | 1.0 | 70162.42 | 8 | | catalog | 1002001.0 | | | 2114110.72 | 380 | | catalog | 1002001.0 | 1.0 | | 348693.97 | 55 | | catalog | 1002001.0 | 1.0 | 1.0 | 76392.12999999999 | 14 | | catalog | 1002001.0 | 1.0 | 2.0 | 118394.33 | 21 | | catalog | 1002001.0 | 1.0 | 4.0 | 29395.79 | 5 | | catalog | 1002001.0 | 1.0 | 5.0 | 35541.97 | 4 | | catalog | 1002001.0 | 1.0 | 6.0 | 26104.36 | 3 | | catalog | 1002001.0 | 1.0 | 9.0 | 18793.969999999998 | 4 | | catalog | 1002001.0 | 1.0 | 10.0 | 44071.42 | 4 | | catalog | 1002001.0 | 2.0 | | 1233961.6999999997 | 225 | | catalog | 1002001.0 | 2.0 | 1.0 | 239511.02000000002 | 51 | | catalog | 1002001.0 | 2.0 | 2.0 | 147993.14 | 26 | | catalog | 1002001.0 | 2.0 | 3.0 | 100086.92999999998 | 17 | | catalog | 1002001.0 | 2.0 | 4.0 | 53524.41999999999 | 13 | | catalog | 1002001.0 | 2.0 | 5.0 | 48494.06 | 10 | | catalog | 1002001.0 | 2.0 | 6.0 | 142857.03999999998 | 20 | | catalog | 1002001.0 | 2.0 | 7.0 | 116557.98 | 16 | | catalog | 1002001.0 | 2.0 | 8.0 | 92743.92999999998 | 24 | | catalog | 1002001.0 | 2.0 | 9.0 | 203943.98999999996 | 38 | | catalog | 1002001.0 | 2.0 | 10.0 | 88249.19 | 10 | | catalog | 1002001.0 | 3.0 | | 91054.32 | 17 | | catalog | 1002001.0 | 3.0 | 2.0 | 25171.130000000005 | 6 | | catalog | 1002001.0 | 3.0 | 7.0 | 27766.7 | 3 | | catalog | 1002001.0 | 3.0 | 8.0 | 38116.49 | 8 | | catalog | 1002001.0 | 4.0 | | 182427.69 | 32 | | catalog | 1002001.0 | 4.0 | 1.0 | 66896.68000000001 | 15 | +---------+------------+------------+---------------+--------------------+------------------+ ``` </details> <details><summary>Result of pretty_format_batches for TPC-DS Q14 in v52</summary> ``` +---------+------------+------------+---------------+--------------------+------------------+ | channel | i_brand_id | i_class_id | i_category_id | sum_sales | sum_number_sales | +---------+------------+------------+---------------+--------------------+------------------+ | | | | | 47788579.86999994 | 11068 | | | | | | 46294358.79000003 | 10609 | | | | | | 40499040.27 | 9321 | | | | | | 37952602.75000005 | 8889 | | | | | | 50256292.020000026 | 11540 | | | | | | 27943616.979999993 | 6397 | | | | | | 43114338.769999966 | 10000 | | | | | | 56239021.03999995 | 13003 | | | | | | 25682800.660000004 | 6012 | | | | | | 38529122.80999996 | 8922 | | | | | | 59222982.15999998 | 13528 | | | | | | 48322926.85999996 | 11228 | | | | | | 39166012.100000024 | 9010 | | | | | | 32661391.260000017 | 7453 | | | | | | 43315152.09999997 | 10008 | | | | | | 37185124.06999998 | 8641 | | catalog | | | | 16671923.720000004 | 3228 | | catalog | | | | 16630833.01 | 3143 | | catalog | | | | 14038550.020000003 | 2798 | | catalog | | | | 13135427.840000002 | 2638 | | catalog | | | | 17604907.44 | 3399 | | catalog | | | | 10119873.490000004 | 1959 | | catalog | | | | 14698922.719999986 | 2919 | | catalog | | | | 19534422.180000022 | 3931 | | catalog | | | | 9075046.950000005 | 1756 | | catalog | | | | 13829338.200000005 | 2662 | | catalog | | | | 21769645.880000025 | 4087 | | catalog | | | | 16890254.589999996 | 3343 | | catalog | | | | 13897305.679999998 | 2680 | | catalog | | | | 11719010.149999999 | 2217 | | catalog | | | | 14773719.710000006 | 2947 | | catalog | | | | 13021675.88999999 | 2615 | | catalog | 1001001.0 | | | 188446.33000000002 | 41 | | catalog | 1001001.0 | | | 53508.79 | 7 | | catalog | 1001001.0 | | | 100105.28 | 23 | | catalog | 1001001.0 | | | 114412.26999999999 | 25 | | catalog | 1001001.0 | | | 77231.70000000001 | 15 | | catalog | 1001001.0 | | | 174489.15000000002 | 42 | | catalog | 1001001.0 | | | 206490.30000000002 | 38 | | catalog | 1001001.0 | | | 45473.85 | 13 | | catalog | 1001001.0 | | | 146344.47 | 27 | | catalog | 1001001.0 | | | 152599.38 | 28 | | catalog | 1001001.0 | | | 206412.37 | 36 | | catalog | 1001001.0 | | | 119368.21 | 23 | | catalog | 1001001.0 | | | 45014.149999999994 | 12 | | catalog | 1001001.0 | | | 50948.8 | 14 | | catalog | 1001001.0 | | | 16883.97 | 3 | | catalog | 1001001.0 | 1.0 | | 100105.28 | 23 | | catalog | 1001001.0 | 1.0 | | 99985.34999999999 | 21 | | catalog | 1001001.0 | 1.0 | | 107555.43000000001 | 23 | | catalog | 1001001.0 | 1.0 | | 161349.39 | 29 | | catalog | 1001001.0 | 1.0 | | 146344.47 | 27 | | catalog | 1001001.0 | 1.0 | | 122521.31000000001 | 25 | | catalog | 1001001.0 | 1.0 | | 77861.85 | 13 | | catalog | 1001001.0 | 1.0 | | 22597.190000000002 | 3 | | catalog | 1001001.0 | 1.0 | | 16883.97 | 3 | | catalog | 1001001.0 | 1.0 | 1.0 | 115019.61 | 20 | | catalog | 1001001.0 | 1.0 | 2.0 | 146344.47 | 27 | | catalog | 1001001.0 | 1.0 | 3.0 | 22597.190000000002 | 3 | | catalog | 1001001.0 | 1.0 | 4.0 | 107555.43000000001 | 23 | | catalog | 1001001.0 | 1.0 | 5.0 | 122521.31000000001 | 25 | | catalog | 1001001.0 | 1.0 | 6.0 | 16883.97 | 3 | | catalog | 1001001.0 | 1.0 | 7.0 | 46329.780000000006 | 9 | | catalog | 1001001.0 | 1.0 | 8.0 | 77861.85 | 13 | | catalog | 1001001.0 | 1.0 | 9.0 | 99985.34999999999 | 21 | | catalog | 1001001.0 | 1.0 | 10.0 | 100105.28 | 23 | | catalog | 1001001.0 | 2.0 | | 68565.38 | 14 | | catalog | 1001001.0 | 2.0 | | 43967.969999999994 | 7 | | catalog | 1001001.0 | 2.0 | | 12633.87 | 3 | | catalog | 1001001.0 | 2.0 | 2.0 | 43967.969999999994 | 7 | | catalog | 1001001.0 | 2.0 | 3.0 | 68565.38 | 14 | | catalog | 1001001.0 | 2.0 | 5.0 | 12633.87 | 3 | | catalog | 1001001.0 | 3.0 | | 60551.64000000001 | 14 | | catalog | 1001001.0 | 3.0 | | 14426.92 | 4 | | catalog | 1001001.0 | 3.0 | | 36821.61 | 7 | | catalog | 1001001.0 | 3.0 | | 30078.07 | 3 | | catalog | 1001001.0 | 3.0 | | 28455.23 | 4 | | catalog | 1001001.0 | 3.0 | | 28351.61 | 11 | | catalog | 1001001.0 | 3.0 | 1.0 | 11100.79 | 5 | | catalog | 1001001.0 | 3.0 | 2.0 | 60551.64000000001 | 14 | | catalog | 1001001.0 | 3.0 | 4.0 | 28455.23 | 4 | | catalog | 1001001.0 | 3.0 | 6.0 | 36821.61 | 7 | | catalog | 1001001.0 | 3.0 | 7.0 | 17250.82 | 6 | | catalog | 1001001.0 | 3.0 | 8.0 | 14426.92 | 4 | | catalog | 1001001.0 | 3.0 | 9.0 | 30078.07 | 3 | | catalog | 1001001.0 | 4.0 | | 47553.200000000004 | 10 | | catalog | 1001001.0 | 4.0 | | 45473.85 | 13 | | catalog | 1001001.0 | 4.0 | | 16558.92 | 8 | | catalog | 1001001.0 | 4.0 | 2.0 | 45473.85 | 13 | | catalog | 1001001.0 | 4.0 | 3.0 | 16558.92 | 8 | | catalog | 1001001.0 | 4.0 | 4.0 | 47553.200000000004 | 10 | | catalog | 1001001.0 | 5.0 | | 29678.5 | 5 | | catalog | 1001001.0 | 5.0 | | 30112.109999999997 | 12 | | catalog | 1001001.0 | 5.0 | 9.0 | 30112.109999999997 | 12 | | catalog | 1001001.0 | 5.0 | 10.0 | 29678.5 | 5 | | catalog | 1001001.0 | 6.0 | | 10261.82 | 3 | | catalog | 1001001.0 | 6.0 | 9.0 | 10261.82 | 3 | | catalog | 1001001.0 | 7.0 | | 18244.94 | 3 | | catalog | 1001001.0 | 7.0 | 7.0 | 18244.94 | 3 | | catalog | 1001001.0 | 8.0 | | 26895.97 | 6 | +---------+------------+------------+---------------+--------------------+------------------+ ``` </details> Not sure yet what's the issue exactly... I'm still triple checking that it's not a problem with datafusion-distributed testing setup -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
