Package: tilemaker Version: 3.0.0-1 Tags: patch
Hi, First ... thanks for maintaining tilemaker in Debian ..I am running tilemaker in a Debian/Trixie based Docker container i built myself every 30 minutes on a pretty small input. I saw occasional crashes of
tilemaker (Every 2-3 days) I added instrumentation to push the coredump to safe store when it happened.With the help of sonnet i found at least 2 allocator races in tilemaker 3.0.0-1 - At least the first one is fixed in current git upstream although the code has changed completely.
As these present crashes i though i at least document a way around in the Debian bug tracker for Debian/Trixie e.g. stable for others
who trigger this.I locally rebuilt the package for the Docker container with these two patches:
This is the first race i found which is also fixed in upstream git.
Description: Lock region mutex when freeing from the thread's own mmap region
destroy() skipped the region mutex for the calling thread's own region,
racing with other threads freeing into it (e.g. BinarySearchNodeStore
deque growth). This corrupts the allocator state and can crash.
Forwarded: no
--- a/src/mmap_allocator.cpp
+++ b/src/mmap_allocator.cpp
@@ -223,6 +223,7 @@
if(mmap_shm_thread_region_ptr != nullptr) {
auto &i = *mmap_shm_thread_region_ptr;
if(p >= (void const *)i.region.data() && p < reinterpret_cast<void
const *>(reinterpret_cast<uint8_t const *>(i.region.data()) + i.region.size())) {
+ std::lock_guard<std::mutex> lock(i.mutex);
allocator_t allocator(i.buffer.get_segment_manager());
return allocator.destroy(reinterpret_cast<uint8_t
*>(p));
}
@@ -231,6 +232,7 @@
if(mmap_file_thread_ptr != nullptr) {
auto &i = *mmap_file_thread_ptr;
if(p >= i.region.get_address() && p < reinterpret_cast<void const
*>(reinterpret_cast<uint8_t const *>(i.region.get_address()) + i.region.get_size())) {
+ std::lock_guard<std::mutex> lock(i.mutex);
allocator_t allocator(i.buffer.get_segment_manager());
allocator.destroy(reinterpret_cast<uint8_t *>(p));
return;
The second crash and fix i found was this - Its still in upstream
tilemaker although the code changed significantly and the race is more
unlikely as upstream changed to std::dequeue - This solution
allocates a static array of 2**24 pointers - This only increases VSZ
when not touching them. Advantage is that no locking on read is
necessary which is missing in 3.0.0-1 and in upstream.
Description: Fix data race on PooledString table vector
tables[tableIndex] was read without a lock while another thread
push_back()ed (reallocating), yielding a garbage pointer in memcpy.
Use a fixed, never-reallocated table instead.
.
Crash (SIGSEGV, tilemaker 3.0.0-1, --input grid.pbf):
.
#0 libc memcpy (vmovdqu %xmm0,(%rdi)), rdi=0x7f6cab91b (garbage)
#1 memcpy (string_fortified.h:29)
#2 PooledString::PooledString(str="L 747 Langenstrasse")
pooled_string.cpp:54 memcpy(tables[tableIndex] + offset, ...)
#3 PooledString::ensureStringIsOwned() pooled_string.cpp:159
#4 AttributePair::ensureStringIsOwned() attribute_store.cpp:65
#5 AttributePairStore::addPair() attribute_store.cpp:181
#6 AttributeStore::addAttribute(key="name") attribute_store.cpp:247
#7 OsmLuaProcessing::Attribute() osm_lua_processing.cpp:831
#8 Lua Attribute() binding (kaguya) osm_lua_processing.cpp:215
#9.. kaguya invoke, liblua5.4, lua_pcallk
.
In the core, tableIndex=0 and tables had size 2 == capacity 2 (just
grown by another thread), so the unlocked read of tables[0] hit the
freed old buffer.
Forwarded: no
--- a/src/pooled_string.cpp
+++ b/src/pooled_string.cpp
@@ -2,9 +2,13 @@
#include <stdexcept>
#include <mutex>
#include <cstring>
+#include <cstdlib>
namespace PooledStringNS {
- std::vector<char*> tables;
+ // Fixed table (never reallocates); lazily committed by the OS.
+ const size_t MaxTables = 1 << 24;
+ char** const tables = (char**)calloc(MaxTables, sizeof(char*));
+ size_t numTables = 0;
std::mutex mutex;
const uint8_t ShortString = 0b00;
@@ -35,8 +39,10 @@
char* buffer = (char*)malloc(spaceLeft);
if (buffer == 0)
throw std::runtime_error("PooledString could not
malloc");
- tables.push_back(buffer);
- tableIndex = tables.size() - 1;
+ if (numTables >= MaxTables)
+ throw std::runtime_error("PooledString out of
tables");
+ tables[numTables] = buffer;
+ tableIndex = numTables++;
}
storage[1] = tableIndex >> 16;
Flo -- Florian Lohoff [email protected] Any sufficiently advanced technology is indistinguishable from magic.
signature.asc
Description: PGP signature

