andygrove commented on code in PR #68:
URL: https://github.com/apache/datafusion-spark/pull/68#discussion_r4607475558
##########
crates/spark/src/function/hash/murmur3.rs:
##########
@@ -0,0 +1,1454 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF)
under one or more contributor license agreements
+
+use std::sync::Arc;
+
+use arrow::array::{ArrayRef, Int32Array, types::ArrowDictionaryKeyType};
+use arrow::buffer::{Buffer, ScalarBuffer};
+use arrow::datatypes::{DataType, Field, FieldRef};
+use datafusion_common::{Result, ScalarValue, internal_err};
+use datafusion_expr::{
+ ColumnarValue, Documentation, ReturnFieldArgs, ScalarFunctionArgs,
ScalarUDFImpl,
+ Signature, Volatility,
+};
+use datafusion_macros::user_doc;
+
+use super::utils::hash_dictionary_values;
+use crate::create_hashes_internal;
+
+const DEFAULT_SEED: u32 = 42;
+
+/// Spark-compatible `hash` expression.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#hash>
+#[user_doc(
+ doc_section(label = "Hashing Functions"),
+ description = "Returns the 32-bit Murmur3 hash (seed 42) of the arguments
as an INT, matching Spark's `hash`. \
+ DataFusion has no built-in equivalent. Arguments are hashed in order,
and each argument's hash seeds the next, \
+ so `hash(a, b)` equals `hash(named_struct('a', a, 'b', b))` and
argument order matters. \
+ NULL arguments, NULL list elements and NULL struct fields are skipped,
so the result is never NULL: \
+ `hash(NULL)` returns 42. The result depends on the input type, as in
Spark: INT hashes 4 bytes and BIGINT 8, \
+ and integer literals are BIGINT in DataFusion but INT in Spark, so
cast them to INT to match Spark's results. \
+ `-0.0` hashes like `0.0`, and every NaN hashes like the canonical NaN.
Dates hash as INT days and \
+ timestamps as BIGINT microseconds, whatever the Arrow unit
(DataFusion's TIMESTAMP literals are \
+ nanoseconds, floored to microseconds). Decimals with precision up to
18 hash as their unscaled long, and wider decimals hash \
+ the minimal big-endian two's complement bytes of the unscaled value.
Lists hash each element in order \
+ (nested lists flatten into the same chain, and an empty list leaves
the seed unchanged), structs hash each field, \
+ and maps hash each key then its value. This crate hashes maps as if
Spark's `spark.sql.legacy.allowHashOnMapType` \
+ were set, where Spark 4 rejects them by default (see
https://github.com/apache/datafusion-spark/issues/69). \
+ Dictionary-encoded values hash like their decoded values. At least one
argument is required.",
+ syntax_example = "hash(expr1[, ..., exprN])",
Review Comment:
Thanks. I kept the hashing as it is. An Interval(MonthDayNano) value with
only days set could also be a real Spark calendar interval, for example from
Comet, and treating it as a day-time interval would give wrong results there.
24cf7ba adds the TIME and interval rules to the hash docs, along with the
INTERVAL '1' DAY case and the arrow_cast workaround. It also adds a test
showing INTERVAL '1' DAY hashes like a zero interval. I also updated the PR
description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]