SebastianGruza commented on code in PR #771:
URL:
https://github.com/apache/hugegraph-toolchain/pull/771#discussion_r4183135576
##########
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/util/DataTypeUtil.java:
##########
@@ -200,7 +203,17 @@ private static Object parseSingleValue(String key, Object
rawValue,
}
} else if (dataType.isUUID()) {
return parseUUID(key, value);
+ } else if (dataType.isDecimal()) {
+ return parseDecimal(key, value);
} else if (dataType.isText()) {
+ if (value instanceof BigDecimal && fromJsonParser(source)) {
Review Comment:
Right. A mapped value that is a BigDecimal on a non-JSON row source is
marked (`MappedDecimal`): a TEXT or id output gets the double text as before
("1.5"), a DECIMAL output keeps every digit. Test
`testMappedFractionOnACsvRow`: 1.50 gives TEXT "1.5", id "1.5" and DECIMAL
1.50; a 20-digit fraction gives the exact DECIMAL and TEXT
"1.2345678901234567E19"; on a JSON row the JSON rule applies as before.
##########
hugegraph-client/src/main/java/org/apache/hugegraph/serializer/BigDecimalSerializer.java:
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hugegraph.serializer;
+
+import java.io.IOException;
+import java.math.BigDecimal;
+
+import org.apache.hugegraph.structure.constant.DataType;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.databind.SerializerProvider;
+import com.fasterxml.jackson.databind.ser.std.StdSerializer;
+
+/**
+ * Write a BigDecimal as a plain JSON number ("1.10", never "1.1E+2").
+ * Jackson's default is the scientific form of BigDecimal.toString(); the
+ * plain form carries every digit, so a server that reads fractions as
+ * BigDecimal (apache/hugegraph#3209) stores a DECIMAL value exactly, and a
+ * server that reads them as double behaves as it always did. The value stays
+ * a number, so numeric keys (DOUBLE, FLOAT, LONG, INT) accept it too.
+ */
+public class BigDecimalSerializer extends StdSerializer<BigDecimal> {
+
+ private static final long serialVersionUID = 1L;
+
+ public BigDecimalSerializer() {
+ super(BigDecimal.class);
+ }
+
+ @Override
+ public void serialize(BigDecimal value, JsonGenerator generator,
+ SerializerProvider provider) throws IOException {
+ generator.writeNumber(exactString(value));
+ }
+
+ /**
+ * The plain form while the value is within the DECIMAL bounds the server
+ * accepts, else the scientific form: still exact and still a JSON number,
+ * but a value such as 1E+999999999 is not expanded into a billion
+ * characters on the client before the server rejects it.
+ */
+ public static String exactString(BigDecimal value) {
+ int scale = value.scale();
+ if (value.precision() <= DataType.DECIMAL_MAX_PRECISION &&
+ scale >= -DataType.DECIMAL_MAX_SCALE && scale <=
DataType.DECIMAL_MAX_SCALE) {
+ return value.toPlainString();
Review Comment:
Right. The plain form is used only when its digit count fits in 128
(precision minus scale for a negative scale); `1E+128` is sent as `1E+128`,
`1E+127` as 128 plain digits. Test in `DecimalDataTypeTest`.
##########
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/util/DataTypeUtil.java:
##########
@@ -328,6 +354,86 @@ private static Number parseNumber(String key, Object value,
}
}
+ private static boolean fromJsonParser(InputSource source) {
+ if (source instanceof FileSource) {
+ return ((FileSource) source).format() == FileFormat.JSON;
+ }
+ if (source instanceof KafkaSource) {
+ return ((KafkaSource) source).getFormat() == FileFormat.JSON;
+ }
+ return false;
+ }
+
+ private static boolean withinDecimalBounds(BigDecimal decimal) {
+ try {
+ DataType.checkDecimalBounds(decimal);
+ return true;
+ } catch (IllegalArgumentException e) {
+ return false;
+ }
+ }
+
+ /**
+ * The string a JSON fraction had before it was read as BigDecimal: the
+ * shortest double representation ("1.5" for 1.50, "1.0E-7" for 1e-7).
+ * TEXT values and mapping lookups keep the keys they always had.
+ */
+ public static String jsonNumberText(BigDecimal value) {
+ return Double.toString(value.doubleValue());
+ }
+
+ /** The lookup key of a field value in the mapping's value map. */
+ public static String mappingKey(Object fieldValue, InputSource source) {
+ if (fieldValue instanceof BigDecimal && fromJsonParser(source)) {
+ return jsonNumberText((BigDecimal) fieldValue);
+ }
+ return String.valueOf(fieldValue);
+ }
+
+ /**
+ * The mapped value when the mapping has an entry for this field value,
+ * else the original value: the lookup key is only a key, so a field
+ * without a mapping entry (e.g. a DECIMAL column of a struct that maps
+ * another field) keeps every digit.
+ */
+ public static Object mapValue(ElementMapping mapping, String fieldName,
Object fieldValue,
+ InputSource source, boolean caseSensitive) {
+ if (mapping.mappingValues().isEmpty()) {
+ return fieldValue;
+ }
+ Object mapped = mapping.mappedValue(fieldName, mappingKey(fieldValue,
source), caseSensitive);
+ return mapped != null ? mapped : fieldValue;
Review Comment:
Right. An unmapped value is `String.valueOf()` again, as before the PR; only
a BigDecimal keeps its object (the round-6 fix). Test: a struct with a
`value_mapping` on `city` and the JSON row `{"active":true}` converts `active`
to "true" for a TEXT key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
