github-actions[bot] commented on code in PR #70205:
URL: https://github.com/apache/doris/pull/70205#discussion_r4600172516
##########
be/src/format/table/es/es_scroll_parser.cpp:
##########
@@ -561,36 +581,36 @@ Status get_date_value_int(const rapidjson::Value& col,
PrimitiveType type, bool
template <PrimitiveType T>
Status get_date_int(const rapidjson::Value& col, PrimitiveType type, bool
pure_doc_value,
typename PrimitiveTypeTraits<T>::CppType* slot,
- const cctz::time_zone& time_zone) {
- // this would happend just only when `enable_docvalue_scan = false`, and
field has timestamp format date from _source
+ const cctz::time_zone& time_zone, std::string_view
date_format) {
if (col.IsNumber()) {
- // ES process date/datetime field would use millisecond timestamp for
index or docvalue
- // processing date type field, if a number is encountered, Doris On ES
will force it to be processed according to ms
- // Doris On ES needs to be consistent with ES, so just divided by 1000
because the unit for from_unixtime is seconds
- return get_date_value_int<T>(col, type, false, slot, time_zone);
+ // ES applies the mapped grammar to source numeric tokens too. Numeric
docvalues are
+ // canonical epoch values; requests from an old FE also retain their
millisecond decoder.
+ const bool mapped_source = !pure_doc_value && !date_format.empty();
+ return get_date_value_int<T>(col, type, mapped_source, slot,
time_zone, date_format);
} else if (col.IsArray() && pure_doc_value && !col.Empty()) {
// this would happened just only when `enable_docvalue_scan = true`
// ES add default format for all field after ES 6.4, if we not
provided format for `date` field ES would impose
// a standard date-format for date field as `2020-06-16T00:00:00.000Z`
// At present, we just process this string format date. After some PR
were merged into Doris, we would impose `epoch_mills` for
// date field's docvalue
if (col[0].IsString()) {
- return get_date_value_int<T>(col[0], type, true, slot, time_zone);
+ return get_date_value_int<T>(col[0], type, true, slot, time_zone,
date_format);
Review Comment:
[P1] Preserve ES 6.x ISO date docvalues with explicit mappings
With the default docvalue scan enabled, a date mapped as `yyyy-MM-dd`
(inferred as DATEV2), or `yyyy-MM-dd HH:mm:ss` (DATETIMEV2), now fails on ES
6.8. `ESScrollQueryBuilder` requests bare `docvalue_fields`; ES 6.x uses its
legacy date-value path and returns ISO text such as `2024-01-01T00:00:00.000Z`,
independently of the source mapping ([ES 6.8 fetch
path](https://github.com/elastic/elasticsearch/blob/v6.8.23/server/src/main/java/org/elasticsearch/search/fetch/subphase/DocValueFieldsFetchSubPhase.java#L81-L142)).
Forwarding the source `date_format` here makes `es_calendar_format_accepts`
reject that string by length/separators before reaching the ISO parser, so
previously working date projections return a cast error. This is distinct from
the existing numeric-source thread: it affects string docvalues. Preserve the
legacy response grammar separately, or request a stable explicit docvalue
format, and cover ISO string docvalues with an explicit mapping context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]