swaminathanmanish commented on code in PR #18976:
URL: https://github.com/apache/pinot/pull/18976#discussion_r3804055004
##########
pinot-plugins/pinot-input-format/pinot-json/src/main/java/org/apache/pinot/plugin/inputformat/json/JSONMessageDecoder.java:
##########
@@ -71,13 +74,24 @@ public void init(Map<String, String> props, Set<String>
fieldsToRead, String top
if (recordExtractorClass == null) {
recordExtractorClass = JSON_RECORD_EXTRACTOR_CLASS;
}
+ String preserveDecimalPrecision = null;
+ if (props != null) {
+ preserveDecimalPrecision =
props.get(PRESERVE_DECIMAL_PRECISION_CONFIG_KEY);
+ }
+
_jsonRecordExtractor =
PluginManager.get().createInstance(recordExtractorClass);
_jsonRecordExtractor.init(fieldsToRead, null);
_fieldsToRead = CollectionUtils.isNotEmpty(fieldsToRead) ?
Set.copyOf(fieldsToRead) : null;
// Direct parsing implements JSONRecordExtractor's conversion contract and
bypasses extract(). Require the
// exact default class so a configured extractor or subclass cannot lose
custom extraction behavior.
_usesDefaultRecordExtractor = _jsonRecordExtractor.getClass() ==
JSONRecordExtractor.class;
- _parser = JsonPayloadFormat.fromConfig(jsonFormat).getParser();
+ JsonPayloadFormat format = JsonPayloadFormat.fromConfig(jsonFormat);
+ _parser = format.getParser();
+ // BigDecimal-preserving parsing goes through Pinot's BigDecimal-aware
text JSON reader, so it applies only
+ // to the TEXT format (the historical default); the binary formats encode
floating point natively.
+ _preserveDecimalPrecision = format == JsonPayloadFormat.TEXT &&
(preserveDecimalPrecision != null
Review Comment:
is this applicable only to json extractor and not to other extractors like
avro/proto etc. I guess the text gets rounded off to double.
##########
pinot-plugins/pinot-input-format/pinot-json/src/main/java/org/apache/pinot/plugin/inputformat/json/JSONMessageDecoder.java:
##########
@@ -52,12 +53,14 @@ public class JSONMessageDecoder implements
StreamMessageDecoder<byte[]> {
private static final String JSON_RECORD_EXTRACTOR_CLASS =
"org.apache.pinot.plugin.inputformat.json.JSONRecordExtractor";
+ private static final String PRESERVE_DECIMAL_PRECISION_CONFIG_KEY =
"preserveDecimalPrecision";
Review Comment:
We are doing opt-in only for backwards compatibility ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
