andygrove commented on code in PR #68:
URL: https://github.com/apache/datafusion-spark/pull/68#discussion_r4607475558


##########
crates/spark/src/function/hash/murmur3.rs:
##########
@@ -0,0 +1,1454 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for 
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) 
under one or more contributor license agreements
+
+use std::sync::Arc;
+
+use arrow::array::{ArrayRef, Int32Array, types::ArrowDictionaryKeyType};
+use arrow::buffer::{Buffer, ScalarBuffer};
+use arrow::datatypes::{DataType, Field, FieldRef};
+use datafusion_common::{Result, ScalarValue, internal_err};
+use datafusion_expr::{
+    ColumnarValue, Documentation, ReturnFieldArgs, ScalarFunctionArgs, 
ScalarUDFImpl,
+    Signature, Volatility,
+};
+use datafusion_macros::user_doc;
+
+use super::utils::hash_dictionary_values;
+use crate::create_hashes_internal;
+
+const DEFAULT_SEED: u32 = 42;
+
+/// Spark-compatible `hash` expression.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#hash>
+#[user_doc(
+    doc_section(label = "Hashing Functions"),
+    description = "Returns the 32-bit Murmur3 hash (seed 42) of the arguments 
as an INT, matching Spark's `hash`. \
+        DataFusion has no built-in equivalent. Arguments are hashed in order, 
and each argument's hash seeds the next, \
+        so `hash(a, b)` equals `hash(named_struct('a', a, 'b', b))` and 
argument order matters. \
+        NULL arguments, NULL list elements and NULL struct fields are skipped, 
so the result is never NULL: \
+        `hash(NULL)` returns 42. The result depends on the input type, as in 
Spark: INT hashes 4 bytes and BIGINT 8, \
+        and integer literals are BIGINT in DataFusion but INT in Spark, so 
cast them to INT to match Spark's results. \
+        `-0.0` hashes like `0.0`, and every NaN hashes like the canonical NaN. 
Dates hash as INT days and \
+        timestamps as BIGINT microseconds, whatever the Arrow unit 
(DataFusion's TIMESTAMP literals are \
+        nanoseconds, floored to microseconds). Decimals with precision up to 
18 hash as their unscaled long, and wider decimals hash \
+        the minimal big-endian two's complement bytes of the unscaled value. 
Lists hash each element in order \
+        (nested lists flatten into the same chain, and an empty list leaves 
the seed unchanged), structs hash each field, \
+        and maps hash each key then its value. This crate hashes maps as if 
Spark's `spark.sql.legacy.allowHashOnMapType` \
+        were set, where Spark 4 rejects them by default (see 
https://github.com/apache/datafusion-spark/issues/69). \
+        Dictionary-encoded values hash like their decoded values. At least one 
argument is required.",
+    syntax_example = "hash(expr1[, ..., exprN])",

Review Comment:
   Thanks. I kept the hashing as it is. An Interval(MonthDayNano) value with 
only days set could also be a real Spark calendar interval, for example from 
Comet, and treating it as a day-time interval would give wrong results there. 
24cf7ba adds the TIME and interval rules to the hash docs, along with the 
INTERVAL '1' DAY case and the arrow_cast workaround. It also adds a test 
showing INTERVAL '1' DAY hashes like a zero interval. I also updated the PR 
description.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to