bitflicker64 commented on code in PR #771:
URL:
https://github.com/apache/hugegraph-toolchain/pull/771#discussion_r4187496453
##########
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/util/DataTypeUtil.java:
##########
@@ -328,6 +358,112 @@ private static Number parseNumber(String key, Object
value,
}
}
+ private static boolean fromJsonParser(InputSource source) {
+ if (source instanceof FileSource) {
+ return ((FileSource) source).format() == FileFormat.JSON;
+ }
+ if (source instanceof KafkaSource) {
+ return ((KafkaSource) source).getFormat() == FileFormat.JSON;
+ }
+ return false;
+ }
+
+ private static boolean withinDecimalBounds(BigDecimal decimal) {
+ try {
+ DataType.checkDecimalBounds(decimal);
+ return true;
+ } catch (IllegalArgumentException e) {
+ return false;
+ }
+ }
+
+ /**
+ * The string a JSON fraction had before it was read as BigDecimal: the
+ * shortest double representation ("1.5" for 1.50, "1.0E-7" for 1e-7).
+ * TEXT values and mapping lookups keep the keys they always had.
+ */
+ public static String jsonNumberText(BigDecimal value) {
+ return Double.toString(value.doubleValue());
+ }
+
+ /** The lookup key of a field value in the mapping's value map. */
+ public static String mappingKey(Object fieldValue, InputSource source) {
+ if (fieldValue instanceof BigDecimal && fromJsonParser(source)) {
+ return jsonNumberText((BigDecimal) fieldValue);
+ }
+ return String.valueOf(fieldValue);
+ }
+
+ /**
+ * The mapped value when the mapping has an entry for this field value,
+ * else the original value: the lookup key is only a key, so a field
+ * without a mapping entry (e.g. a DECIMAL column of a struct that maps
+ * another field) keeps every digit.
+ */
+ public static Object mapValue(ElementMapping mapping, String fieldName,
Object fieldValue,
+ InputSource source, boolean caseSensitive) {
+ if (mapping.mappingValues().isEmpty()) {
+ return fieldValue;
+ }
+ Object mapped = mapping.mappedValue(fieldName, mappingKey(fieldValue,
source), caseSensitive);
+ if (mapped != null) {
+ // a fractional replacement value comes from the struct's JSON as
+ // a BigDecimal: on a non-JSON row source it is marked so that a
+ // TEXT or id output gets the double text the struct gave before
+ // ("1.5"), while a DECIMAL output keeps every digit
+ return mapped instanceof BigDecimal && !fromJsonParser(source)
+ ? new MappedDecimal((BigDecimal) mapped) : mapped;
Review Comment:
Important: On a non-JSON source, `MappedDecimal` goes everywhere
`mappingValue()` output goes, but only `convert()` and `idText()` unwrap it.
`ElementBuilder.customizeId` calls `DataTypeUtil.parseNumber(idField, idValue)`
for CUSTOMIZE_NUMBER ids (lines 357 and 843, and also for a short-id config
with a numeric `id_field_type`). That method accepts only a `Number` or a
`String`, so a `MappedDecimal` gets an IllegalArgumentException and every row
with that value is rejected.
At the base, the struct's replacement was a `Double` and the id loaded. A
probe against the compiled head with `value_mapping: {"code": {"A": 1.0}}` on a
CSV `FileSource` shows the difference:
```
mapped class: MappedDecimal
CUSTOMIZE_NUMBER id threw: The value(key='code') must can be casted to Long,
but got '1.0'(...DataTypeUtil$MappedDecimal)
base (Double 1.0) id: 1
```
A DATE key fed by the same mapping changes too: it used to get `Date(1)`
from the Double and now gets the string "1.0", which the date format rejects.
Please unwrap `MappedDecimal` in `parseNumber(String, Object)`, either by
passing its `value` on as a `Number` or by making `MappedDecimal` extend
`Number`. Also add a `NullValuesTest` case that builds a CUSTOMIZE_NUMBER id
from a mapped fraction on a CSV row.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
