mbutrovich commented on code in PR #68:
URL: https://github.com/apache/datafusion-spark/pull/68#discussion_r4605419943


##########
crates/spark/src/function/hash/murmur3.rs:
##########
@@ -0,0 +1,1454 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for 
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) 
under one or more contributor license agreements
+
+use std::sync::Arc;
+
+use arrow::array::{ArrayRef, Int32Array, types::ArrowDictionaryKeyType};
+use arrow::buffer::{Buffer, ScalarBuffer};
+use arrow::datatypes::{DataType, Field, FieldRef};
+use datafusion_common::{Result, ScalarValue, internal_err};
+use datafusion_expr::{
+    ColumnarValue, Documentation, ReturnFieldArgs, ScalarFunctionArgs, 
ScalarUDFImpl,
+    Signature, Volatility,
+};
+use datafusion_macros::user_doc;
+
+use super::utils::hash_dictionary_values;
+use crate::create_hashes_internal;
+
+const DEFAULT_SEED: u32 = 42;
+
+/// Spark-compatible `hash` expression.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#hash>
+#[user_doc(
+    doc_section(label = "Hashing Functions"),
+    description = "Returns the 32-bit Murmur3 hash (seed 42) of the arguments 
as an INT, matching Spark's `hash`. \
+        DataFusion has no built-in equivalent. Arguments are hashed in order, 
and each argument's hash seeds the next, \
+        so `hash(a, b)` equals `hash(named_struct('a', a, 'b', b))` and 
argument order matters. \
+        NULL arguments, NULL list elements and NULL struct fields are skipped, 
so the result is never NULL: \
+        `hash(NULL)` returns 42. The result depends on the input type, as in 
Spark: INT hashes 4 bytes and BIGINT 8, \
+        and integer literals are BIGINT in DataFusion but INT in Spark, so 
cast them to INT to match Spark's results. \
+        `-0.0` hashes like `0.0`, and every NaN hashes like the canonical NaN. 
Dates hash as INT days and \
+        timestamps as BIGINT microseconds, whatever the Arrow unit 
(DataFusion's TIMESTAMP literals are \
+        nanoseconds, floored to microseconds). Decimals with precision up to 
18 hash as their unscaled long, and wider decimals hash \
+        the minimal big-endian two's complement bytes of the unscaled value. 
Lists hash each element in order \
+        (nested lists flatten into the same chain, and an empty list leaves 
the seed unchanged), structs hash each field, \
+        and maps hash each key then its value. This crate hashes maps as if 
Spark's `spark.sql.legacy.allowHashOnMapType` \
+        were set, where Spark 4 rejects them by default (see 
https://github.com/apache/datafusion-spark/issues/69). \
+        Dictionary-encoded values hash like their decoded values. At least one 
argument is required.",
+    syntax_example = "hash(expr1[, ..., exprN])",

Review Comment:
   The description doesn't say how `TIME` and interval values hash, and one 
case will surprise users. DataFusion types the literal `INTERVAL '1' DAY` as 
`Interval(MonthDayNano)`, so `hash(INTERVAL '1' DAY)` here returns 
`1081828689`. That is the calendar-interval arm, which leaves out the days, so 
it's the hash of a zero interval. Spark 4.2.0 returns `892267096`, because its 
`INTERVAL '1' DAY` is a day-time interval hashed as a BIGINT.
   
   Could the description say that `TIME` hashes as BIGINT nanoseconds since 
midnight, `Interval(YearMonth)` as INT months, and `Duration(Microsecond)` as 
BIGINT microseconds? It could also say that `Interval(MonthDayNano)` hashes the 
microseconds and then the months but not the days, as Spark 4.2.0's [generated 
code](https://github.com/apache/spark/blob/32f7299601108917fb01920a54e084595b7b3bf8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala#L428-L431)
 does, so a DataFusion `INTERVAL '<n>' DAY` literal hashes like a zero 
interval. Spark 4.3 includes the days (SPARK-58236), which matters to engines 
that target more than one Spark release. The cases at the end of `hash.slt` 
already cover each of these.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to