Hi Hive, We have tested and put up a patch for this: https://issues.apache.org/jira/browse/HIVE-23133 it is done pretty much the same as in Spark: https://github.com/apache/spark/pull/25279 , it could be a very good fix for running Hive across different platforms.
BR, On Fri, Apr 3, 2020 at 3:18 PM Zhenyu Zheng <[email protected]> wrote: > Hi Hive, > > Currently, we have set up an ARM CI to test out how Hive works on ARM > platform: > > https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/ > > Among the failures, we have observed that some numeric operations can have > different result across hardware archs, such as: > > > https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/ > > > https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/ > > > https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/ > > we can see that the calculation results of log, exp, cos, toRadians etc is > slitly different than the .out file results that we are > > comparing(they are tested and wrote on X86 machines), this is because of > we use Math Library > <https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html> for these > kind of calculations. > > and according to the illustration > <https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html>: > > *Unlike some of the numeric methods of class StrictMath, all > implementations of the equivalent functions of class Math are not* > *defined to return the bit-for-bit same results. This relaxation permits > better-performing implementations where strict reproducibility* > *is not required.* > > *By default many of the Math methods simply call the equivalent method in > StrictMath for their implementation.* > *Code generators are encouraged to use platform-specific native libraries > or microprocessor instructions, where available,* > *to provide higher-performance implementations of Math methods.* > > so the result will have difference across hardware archs. > > On the other hand, JAVA provided another library StrictMath > <https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html> > that will not have this kind of problem as according to its' reference > <https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html>: > > *To help ensure portability of Java programs, the definitions of some of > the numeric functions in this package require that they produce* > *the same results as certain published algorithms.* > > So in order to fix the above mentioned problem, we have to consider switch > to use StrictMath instead of Math. > > I have also reported an issue about this > https://issues.apache.org/jira/browse/HIVE-23133 so comments and > suggestions are welcome. > > > BR, > > Zhenyu Zheng > > >
