github-actions[bot] commented on code in PR #70552:
URL: https://github.com/apache/doris/pull/70552#discussion_r4652843770
##########
be/src/exec/operator/group_commit_block_sink_operator.cpp:
##########
@@ -408,6 +502,9 @@ Status
GroupCommitBlockSinkOperatorX::sink_impl(RuntimeState* state, Block* inpu
rows_to_keep.data() +
rows_to_keep.size()));
block->swap(res_block.to_block());
}
+ // Check retained rows before buffering/enqueueing: async success only
guarantees WAL storage,
+ // and an invalid vector discovered by the segment writer cannot be
repaired by WAL replay.
+ RETURN_IF_ERROR(_validate_ann_vectors(*block));
Review Comment:
[P1] Preserve request atomicity when a later ANN block is rejected
For a request whose valid prefix exceeds
`group_commit_memory_rows_for_max_filter_ratio` (10,000 rows by default),
`_add_block()` has already flushed earlier blocks to the shared queue, and
async mode has written them to WAL. If a later block has the wrong vector
dimension, this return fails only the submitting request. `close()` propagates
the error and the local-state destructor merely calls `remove_load_id()`; that
removes the dependency without retracting queued/WAL data. The separate
background scan can therefore finish successfully and commit the valid prefix
of a request reported as failed. Previously the malformed block reached the
segment writer and failed that transaction. Validate/stage the whole request
before exposing its blocks, or provide request-scoped rollback; cancelling the
shared queue would also affect other accepted loads. The small validator/load
tests miss the multi-block case beyond the flush threshold.
##########
be/src/exprs/function/array/function_array_distance.h:
##########
@@ -109,74 +111,69 @@ class FunctionArrayDistance : public IFunction {
"Arguments for function {} must be arrays",
get_name());
}
- return std::make_shared<DataTypeFloat32>();
+ return make_nullable(std::make_shared<DataTypeFloat32>());
}
- // All array distance functions has always not nullable return type.
- // We want to make sure throw exception if input columns contain NULL.
+ // Handle outer NULLs explicitly so their nested array payload is never
inspected.
bool use_default_implementation_for_nulls() const override { return false;
}
- // Validate that neither outer column nor inner array elements contain
NULL.
- // Distance functions always throw on NULL input.
- static void _validate_no_nulls(const ColumnPtr& col, const char* arg_name,
- const String& func_name) {
- const IColumn* raw = col.get();
-
- // Unwrap const
- if (is_column_const(*raw)) {
- raw = assert_cast<const
ColumnConst*>(raw)->get_data_column_ptr().get();
- }
-
- // Check outer nullable
- if (const auto* nullable_raw =
check_and_get_column<ColumnNullable>(raw)) {
- if (raw->has_null()) {
- throw doris::Exception(ErrorCode::INVALID_ARGUMENT,
- "{} for function {} cannot be null",
arg_name, func_name);
+ static Status _validate_array_elements(const ColumnArrayView<TYPE_FLOAT>&
view, size_t num_rows,
+ const char* arg_name) {
+ for (size_t row = 0; row < num_rows; ++row) {
+ if (view.is_null_at(row)) {
+ // The nested payload of a NULL data row is not a vector.
+ continue;
+ }
+ const auto array = view[row];
+ if (array.size() == 0) {
+ continue;
+ }
+ if (std::memchr(array.get_null_map_data(), 1, array.size()) !=
nullptr) {
+ return Status::InvalidArgument("{} for function {} cannot have
null", arg_name,
+ name);
}
- raw = nullable_raw->get_nested_column_ptr().get();
- }
-
- // Check inner nullable (array elements)
- const auto& array_col = assert_cast<const ColumnArray&>(*raw);
- if (is_column_nullable(*array_col.get_data_ptr()) &&
array_col.get_data_ptr()->has_null()) {
- throw doris::Exception(ErrorCode::INVALID_ARGUMENT,
- "{} for function {} cannot have null",
arg_name, func_name);
}
+ return Status::OK();
}
- Status execute_impl(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
+ Status execute_impl(FunctionContext*, Block& block, const ColumnNumbers&
arguments,
uint32_t result, size_t input_rows_count) const
override {
const auto& col1 = block.get_by_position(arguments[0]).column;
const auto& col2 = block.get_by_position(arguments[1]).column;
- // Validate no NULLs (distance functions always throw on NULL input)
- _validate_no_nulls(col1, "First argument", get_name());
- _validate_no_nulls(col2, "Second argument", get_name());
-
- // Create views — handles Const/Nullable unwrapping automatically
auto view1 = ColumnArrayView<TYPE_FLOAT>::create(col1);
auto view2 = ColumnArrayView<TYPE_FLOAT>::create(col2);
+ RETURN_IF_ERROR(_validate_array_elements(view1, view1.is_const ? 1 :
input_rows_count,
+ "First argument"));
+ RETURN_IF_ERROR(_validate_array_elements(view2, view2.is_const ? 1 :
input_rows_count,
+ "Second argument"));
auto dst = ColumnType::create(input_rows_count);
auto& dst_data = dst->get_data();
+ auto null_map = ColumnUInt8::create(input_rows_count, 0);
for (size_t row = 0; row < input_rows_count; ++row) {
+ // Missing vectors have no distance, including NULL/NULL and []/[].
+ if (view1.is_null_at(row) || view2.is_null_at(row) ||
view1[row].size() == 0 ||
+ view2[row].size() == 0) {
+ null_map->get_data()[row] = 1;
+ dst_data[row] = 0;
+ continue;
+ }
auto a1 = view1[row];
auto a2 = view2[row];
- const float* p1 = a1.get_data();
- const float* p2 = a2.get_data();
- auto dim1 = a1.size();
- auto dim2 = a2.size();
-
+ const auto dim1 = a1.size();
+ const auto dim2 = a2.size();
if (dim1 != dim2) [[unlikely]] {
return Status::InvalidArgument(
"function {} have different input element sizes of
array: {} and {}",
get_name(), dim1, dim2);
}
- dst_data[row] = DistanceImpl::distance(p1, p2, dim1);
+ dst_data[row] = DistanceImpl::distance(a1.get_data(),
a2.get_data(), dim1);
}
- block.replace_by_position(result, std::move(dst));
+ block.replace_by_position(result,
Review Comment:
[P1] Honor older FE result metadata during a rolling upgrade
An old FE declares these array-distance results as nonnullable FLOAT,
including an ordinary nonconstant query such as `SELECT l2_distance(vec,[0,0])
FROM t` over valid, nonempty vectors. A newly upgraded BE now unconditionally
constructs `ColumnNullable(Float32)`. Function construction accepts the
nested-type match, but `VExpr::execute_column()` checks the physical column
against the FE-declared `Float32` and returns `return column type mismatch`
before result serialization. This affects existing queries even while
`enable_ann_index_empty_array=false`; the default-off write option provides no
compatibility protection. Add a result-metadata/execution-version compatibility
path so old FE plans continue to receive the legacy physical type until the
cluster can use the new nullable contract. Requiring a coordinated FE/BE
upgrade in the description does not preserve rolling query compatibility.
##########
be/src/storage/index/ann/faiss_ann_index.cpp:
##########
@@ -888,30 +997,71 @@ doris::Status FaissVectorIndex::range_search(const float*
query_vec, const float
}
}
+ // The pinned FAISS IDMap::range_search slices SearchParametersHNSW to its
base
+ // type. Translate the selector here and call the child with the original
params,
+ // preserving efSearch/bounded_queue even when filtering out segment
rowids.
+ auto* mapped = dynamic_cast<faiss::IndexIDMap*>(_index.get());
+ faiss::Index* search_index = mapped != nullptr ? mapped->index :
_index.get();
+ std::unique_ptr<faiss::IDSelectorTranslated> translated_selector;
+ if (mapped != nullptr && search_param->sel != nullptr) {
+ translated_selector =
+ std::make_unique<faiss::IDSelectorTranslated>(mapped->id_map,
search_param->sel);
+ search_param->sel = translated_selector.get();
+ }
+ // Empty arrays must not reappear when an L2 > radius / IP < radius
predicate
+ // complements the hits. E.g. [] is absent from both sides of a distance
comparison.
+ roaring::Roaring sparse_candidates;
+ const roaring::Roaring* range_domain = params.roaring;
+ // Dense files already cover every data row. Avoid collecting all IVF list
IDs
+ // (and reading every on-disk list) just to reconstruct the same candidate
domain.
+ if (needs_complement && cast_set<size_t>(_index->ntotal) !=
params.rows_of_segment) {
+ SCOPED_RAW_TIMER(&result.engine_prepare_ns);
+ ScopedIoCtxBinding io_ctx_binding(params.io_ctx);
+ RETURN_IF_ERROR(_init_indexed_rows());
+ sparse_candidates = *params.roaring & *_indexed_rows;
+ range_domain = &sparse_candidates;
+ }
+
faiss::RangeSearchResult native_search_result(1, true);
{
// Engine search: FAISS range_search
SCOPED_RAW_TIMER(&result.engine_search_ns);
ScopedIoCtxBinding io_ctx_binding(params.io_ctx);
try {
+ // FAISS uses strict comparisons: squared L2 < radius and IP >
radius.
+ // Include equality by moving the bound one float toward the
outside.
+ // Complementing flips inclusiveness: L2 > 3 must exclude L2 <= 3.
+ const bool include_boundary = params.include_boundary !=
needs_complement;
if (_metric == AnnIndexMetric::L2) {
- if (radius <= 0) {
- _index->range_search(1, query_vec, 0.0f,
&native_search_result,
- search_param.get());
- } else {
- _index->range_search(1, query_vec, radius * radius,
&native_search_result,
- search_param.get());
+ float search_radius = radius > 0 ? radius * radius : 0.0F;
+ // Negative L2 bounds have no matching distances, including at
zero.
+ if (include_boundary && radius >= 0) {
Review Comment:
[P1] Preserve the original DOUBLE threshold when adjusting range bounds
With an L2 index and stored vector `[1,0]`, the predicate
`CAST(l2_distance_approximate(vec,[0,0]) AS DOUBLE) <= CAST(0.99999999 AS
DOUBLE)` must reject the row: its distance is exactly 1.
`prepare_ann_range_search()` accepts this cast/literal shape but narrows the
bound to FLOAT 1. This new inclusive adjustment then searches squared distances
below `nextafter(1,+inf)`, so it incorrectly includes that row; the previous
strict bound excluded it. `VExprContext::evaluate_ann_range_search()` installs
the bitmap as the original comparison result, so the DOUBLE predicate is not
rechecked. The symmetric IP strict/complement paths also change results near
rounded bounds. Retain the original threshold and derive operator-aware bounds,
or use scalar fallback when translation is inexact; cover DOUBLE thresholds
immediately above/below representable distances as well as exact integer
boundaries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]