luocooong commented on pull request #2299: URL: https://github.com/apache/drill/pull/2299#issuecomment-905316748
@paul-rogers Thank you for the question. In short, the mongo storage plugin of Drill supported both the BSON loader and JSON loader. The BSON loader is the default handler, unless user set the `store.mongo.bson.record.reader` to `false` (In this case, Drill will use the JSON loader to parse the JSON string). https://github.com/apache/drill/blob/7d4d75a9bb9b4b697ef5bab74c2156810e31dc9b/contrib/storage-mongo/src/main/java/org/apache/drill/exec/store/mongo/MongoRecordReader.java#L197-L204 We see that mongo driver will convert the data structure to JSON string (enabled the JSON loader). In the mongo-spec, the driver supported two JSON text output format : `Canonical Format` and `Relaxed Format`. Notably, the Java driver use the `Relaxed` format as output when call the `cursor.next().toJson()`. So we don't see both of the following date format at the same time. ```java // Canonical {"$date":{"$numberLong":"1565546054692"}} // Relaxed {"$date":"2019-08-11T17:54:14.692Z"} ``` If application need the `Canonical` format, the mongo reader code is that : ```java JsonWriterSettings settings = JsonWriterSettings.builder().outputMode(JsonMode.EXTENDED).build(); String doc = cursor.next().toJson(settings); ``` It is showed that the JSON output format is depending on the application. ```java // Canonical output {"_id": {"$oid": "61226ac33b29ff6306b36aa9"}, "date": {"$date": {"$numberLong": "1629645507258"}}} // Relaxed output {"_id": {"$oid": "61226ac33b29ff6306b36aa9"}, "date": {"$date": "2021-08-22T15:18:27.258Z"}} ``` I think I've replied the first three items. Once the toJson() is done, the next operation (parse & save to vector value) is no difference than query the JSON files. The only issue is that we are now using an old version of JSON loader, the new JSON loader contains more complete unit tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
