github-actions[bot] commented on code in PR #70205:
URL: https://github.com/apache/doris/pull/70205#discussion_r4600172516


##########
be/src/format/table/es/es_scroll_parser.cpp:
##########
@@ -561,36 +581,36 @@ Status get_date_value_int(const rapidjson::Value& col, 
PrimitiveType type, bool
 template <PrimitiveType T>
 Status get_date_int(const rapidjson::Value& col, PrimitiveType type, bool 
pure_doc_value,
                     typename PrimitiveTypeTraits<T>::CppType* slot,
-                    const cctz::time_zone& time_zone) {
-    // this would happend just only when `enable_docvalue_scan = false`, and 
field has timestamp format date from _source
+                    const cctz::time_zone& time_zone, std::string_view 
date_format) {
     if (col.IsNumber()) {
-        // ES process date/datetime field would use millisecond timestamp for 
index or docvalue
-        // processing date type field, if a number is encountered, Doris On ES 
will force it to be processed according to ms
-        // Doris On ES needs to be consistent with ES, so just divided by 1000 
because the unit for from_unixtime is seconds
-        return get_date_value_int<T>(col, type, false, slot, time_zone);
+        // ES applies the mapped grammar to source numeric tokens too. Numeric 
docvalues are
+        // canonical epoch values; requests from an old FE also retain their 
millisecond decoder.
+        const bool mapped_source = !pure_doc_value && !date_format.empty();
+        return get_date_value_int<T>(col, type, mapped_source, slot, 
time_zone, date_format);
     } else if (col.IsArray() && pure_doc_value && !col.Empty()) {
         // this would happened just only when `enable_docvalue_scan = true`
         // ES add default format for all field after ES 6.4, if we not 
provided format for `date` field ES would impose
         // a standard date-format for date field as `2020-06-16T00:00:00.000Z`
         // At present, we just process this string format date. After some PR 
were merged into Doris, we would impose `epoch_mills` for
         // date field's docvalue
         if (col[0].IsString()) {
-            return get_date_value_int<T>(col[0], type, true, slot, time_zone);
+            return get_date_value_int<T>(col[0], type, true, slot, time_zone, 
date_format);

Review Comment:
   [P1] Preserve ES 6.x ISO date docvalues with explicit mappings
   
   With the default docvalue scan enabled, a date mapped as `yyyy-MM-dd` 
(inferred as DATEV2), or `yyyy-MM-dd HH:mm:ss` (DATETIMEV2), now fails on ES 
6.8. `ESScrollQueryBuilder` requests bare `docvalue_fields`; ES 6.x uses its 
legacy date-value path and returns ISO text such as `2024-01-01T00:00:00.000Z`, 
independently of the source mapping ([ES 6.8 fetch 
path](https://github.com/elastic/elasticsearch/blob/v6.8.23/server/src/main/java/org/elasticsearch/search/fetch/subphase/DocValueFieldsFetchSubPhase.java#L81-L142)).
 Forwarding the source `date_format` here makes `es_calendar_format_accepts` 
reject that string by length/separators before reaching the ISO parser, so 
previously working date projections return a cast error. This is distinct from 
the existing numeric-source thread: it affects string docvalues. Preserve the 
legacy response grammar separately, or request a stable explicit docvalue 
format, and cover ISO string docvalues with an explicit mapping context.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to