PragmaTwice opened a new issue, #2481: URL: https://github.com/apache/kvrocks/issues/2481
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/kvrocks/issues) and found no similar issues. ### Motivation We can try to use this repo: https://github.com/qdrant/vector-db-benchmark After some simple patching: ```patch diff --git a/engine/clients/redis/configure.py b/engine/clients/redis/configure.py index a5e6fe8..b2b8af0 100644 --- a/engine/clients/redis/configure.py +++ b/engine/clients/redis/configure.py @@ -7,6 +7,7 @@ from redis.commands.search.field import ( TextField, VectorField, ) +from redis.commands.search.indexDefinition import IndexDefinition, IndexType from benchmark.dataset import Dataset from engine.base_client.configure import BaseConfigurator @@ -81,7 +82,7 @@ class RedisConfigurator(BaseConfigurator): name="vector", algorithm="HNSW", attributes={ - "TYPE": "FLOAT32", + "TYPE": "FLOAT64", "DIM": dataset.config.vector_size, "DISTANCE_METRIC": self.DISTANCE_MAPPING[dataset.config.distance], **self.collection_params.get("hnsw_config", {}), @@ -97,8 +98,13 @@ class RedisConfigurator(BaseConfigurator): ] for conn in conns: search_namespace = conn.ft() + idef = IndexDefinition( + prefix=[''], + index_type=IndexType.HASH + ) + idef.args = idef.args[:-2] try: - search_namespace.create_index(fields=index_fields) + search_namespace.create_index(fields=index_fields, definition=idef) except redis.ResponseError as e: if "Index already exists" not in str(e): raise e diff --git a/engine/clients/redis/upload.py b/engine/clients/redis/upload.py index cd4b888..0e29533 100644 --- a/engine/clients/redis/upload.py +++ b/engine/clients/redis/upload.py @@ -57,7 +57,7 @@ class RedisUploader(BaseUploader): cls.client.hset( str(idx), mapping={ - "vector": np.array(vec).astype(np.float32).tobytes(), + "vector": np.array(vec).astype(np.float64).tobytes(), **payload, **geopoints, }, ``` We can start a kvrocks instance and run this benchmark: ```bash python3 run.py --engines redis-default --datasets 'glove-25-angular' ``` cc @Beihao-Zhou ### Solution Currently we can get some coredumps in the vector indexing phase, e.g.: ``` E20240809 14:55:31.865105 134 signal_util.h:36] Stack trace (most recent call first): #0 0x0000555f0502cac9 in SegvHandler at /usr/bin/kvrocks #1 0x00007f2e599c004f in __sigaction at /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007f2e59a0ee2c in pthread_key_delete at /lib/x86_64-linux-gnu/libc.so.6 #3 0x00007f2e599bffb1 in gsignal at /lib/x86_64-linux-gnu/libc.so.6 #4 0x00007f2e599aa471 in abort at /lib/x86_64-linux-gnu/libc.so.6 #5 0x0000555f04f77917 in __gnu_cxx::__verbose_terminate_handler() [clone .cold] at vterminate.o #6 0x0000555f05b16c09 in __cxxabiv1::__terminate(void (*)()) at /usr/bin/kvrocks #7 0x0000555f05b16c74 in std::terminate() at /usr/bin/kvrocks #8 0x0000555f05b16dc7 in __cxa_throw at /usr/bin/kvrocks #9 0x0000555f04f7908d in std::__throw_bad_alloc() at /usr/bin/kvrocks #10 0x0000555f04f77509 in handleOOM(unsigned long, bool) [clone .cold] at /kvrocks/build/_deps/jemalloc-src/src/jemalloc_cpp.cpp:90 #11 0x0000555f051d6e4a in redis::VectorItem::VectorItem(redis::VectorItem const&) at /usr/bin/kvrocks #12 0x0000555f051e561a in redis::IndexUpdater::UpdateHnswVectorIndex(std::basic_string_view<char, std::char_traits<char> >, kqir::Value const&, kqir::Value const&, redis::SearchKey const&, redis::HnswVectorFieldMetadata*) const at /usr/bin/kvrocks #13 0x0000555f051ec7be in redis::IndexUpdater::Update(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, kqir::Value, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, kqir::Value> > > const&, std::basic_string_view<char, std::char_traits<char> >) const at /usr/bin/kvrocks #14 0x0000555f05228549 in redis::Connection::ExecuteCommands(std::deque<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >*) at /usr/bin/kvrocks #15 0x0000555f0505db30 in EvbufCallbackBase<redis::Connection, true, true, true>::readCB(bufferevent*, void*) at /usr/bin/kvrocks #16 0x0000555f05978a0d in bufferevent_run_deferred_callbacks_unlocked at bufferevent.c #17 0x0000555f059808a3 in event_process_active_single_queue at event.c #18 0x0000555f05980f3e in event_process_active at event.c #19 0x0000555f05982cf8 in event_base_loop.constprop.0 at event.c #20 0x0000555f0523d46d in std::thread::_State_impl<std::thread::_Invoker<std::tuple<util::CreateThread<WorkerThread::Start()::{lambda()#1}>(char const*, WorkerThread::Start()::{lambda()#1})::{lambda()#1}> > >::_M_run() at /usr/bin/kvrocks #21 0x0000555f05b96312 in execute_native_thread_routine at thread.o #22 0x00007f2e59a0d133 in pthread_condattr_setpshared at /lib/x86_64-linux-gnu/libc.so.6 #23 0x00007f2e59a8ca3f in __clone at /lib/x86_64-linux-gnu/libc.so.6 ``` Maybe we can try to solve it first. ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
