Hi, On 2014-05-06 13:56:41 +0200, Andres Freund wrote: > On 2014-05-05 23:20:43 -0400, Robert Haas wrote: > > On Mon, May 5, 2014 at 6:54 PM, Tom Lane <t...@sss.pgh.pa.us> wrote: > > > I'm not confident that it'll be useful either. But I am confident that > > > if we don't put it in now, and decide we want it later, there will be > > > complaints when we change the API. Better to have an ignored parameter > > > than no parameter. > > > > I'm generally skeptical of that philosophy. If we put in an ignored > > parameter, people may pass pointers to NULL or to garbage or to an > > overly-long string, and they won't know it's broken until we make it > > do something; at which point their code will begin to fail without > > warning. > > If it were a complex change, maybe. But I don't think that's likely > here. > Assert(name != NULL && strlen(name) > 0 && strlen(name) < NAMEDATALEN); > should perfectly do the trick.
Attached are two patches: a) Patch addin a 'name' parameter to dsm_create(). I think we should apply this to 9.4. b) pg_dynamic_shmem_allocations and pg_static_shmem_allocations views. The previous version didn't include dsm support and didn't take the required lock. I am not so sure whether b) should be applied together with a) in 9.4, but I'd be happy enough to add docs if people agree with the naming. FWIW, I like dsm_create()'s internals more after this patch... postgres=# \d pg_dynamic_shmem_allocations View "pg_catalog.pg_dynamic_shmem_allocations" Column | Type | Modifiers --------+--------+----------- handle | bigint | name | text | size | bigint | refcnt | bigint | postgres=# \d pg_static_shmem_allocations View "pg_catalog.pg_static_shmem_allocations" Column | Type | Modifiers -----------+---------+----------- key | text | off | bigint | size | bigint | allocated | boolean | postgres=# SELECT * FROM pg_dynamic_shmem_allocations; handle | name | size | refcnt ------------+-------------+-------+-------- 1120921036 | test_shm_mq | 65656 | 1 (1 row) postgres=# SELECT * FROM pg_static_shmem_allocations ORDER BY key NULLS FIRST; key | off | size | allocated -------------------------------------+------------+------------+----------- | 2222605024 | 1727776 | f | | 34844752 | t Async Ctl | 2222539168 | 65856 | t Async Queue Control | 2222537784 | 1384 | t AutoVacuum Data | 2222533576 | 224 | t Backend Activity Buffer | 2217099552 | 114688 | t Backend Application Name Buffer | 2217085216 | 7168 | t Backend Client Host Name Buffer | 2217092384 | 7168 | t Backend Status Array | 2217061024 | 24192 | t Background Worker Data | 2217214256 | 1992 | t BTree Vacuum State | 2222535768 | 1356 | t Buffer Blocks | 51365312 | 2147483648 | t Buffer Descriptors | 34588096 | 16777216 | t Buffer Strategy Status | 2213546176 | 32 | t Checkpointer Data | 2217290656 | 5242920 | t CLOG Ctl | 33601152 | 525312 | t Control File | 16796384 | 240 | t Fast Path Strong Relation Lock Data | 2214767072 | 4100 | t FinishedSerializableTransactions | 2216841952 | 16 | t LOCK hash | 2213546208 | 2160 | t MultiXactMember Ctl | 34455488 | 131648 | t MultiXactOffset Ctl | 34389632 | 65856 | t OldSerXidControlData | 2216973632 | 16 | t OldSerXid SLRU Ctl | 2216841984 | 131648 | t PMSignalState | 2217285400 | 940 | t PREDICATELOCK hash | 2215182944 | 2160 | t PREDICATELOCKTARGET hash | 2214771176 | 2160 | t PredXactList | 2216348384 | 88 | t Prepared Transaction Table | 2217214240 | 16 | t Proc Array | 2217060536 | 488 | t Proc Header | 2216973648 | 88 | t PROCLOCK hash | 2214183264 | 2160 | t ProcSignalSlots | 2217286344 | 4284 | t RWConflictPool | 2216573120 | 24 | t SERIALIZABLEXID hash | 2216518720 | 2160 | t Shared Buffer Lookup Table | 2198848960 | 16496 | t Shared MultiXact State | 34587136 | 936 | t shmInvalBuffer | 2217216256 | 69144 | t SUBTRANS Ctl | 34126464 | 263168 | t Sync Scan Locations List | 2222537128 | 656 | t Wal Receiver Ctl | 2222534576 | 1192 | t Wal Sender Ctl | 2222533800 | 776 | t XLOG Ctl | 16796640 | 16804496 | t (43 rows) Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
>From 43ae2a5397fba3b83afced6ec813449a1c87f8c0 Mon Sep 17 00:00:00 2001 From: Andres Freund <and...@anarazel.de> Date: Tue, 6 May 2014 19:42:36 +0200 Subject: [PATCH 1/2] Associate names to created dynamic shared memory segments. At some later point we want to add a view show all allocated dynamic shared memory segments so admins can understand resource usage. To avoid breaking the API in 9.5 add the necessary name now. --- contrib/test_shm_mq/setup.c | 2 +- src/backend/storage/ipc/dsm.c | 60 ++++++++++++++++++++++++++----------------- src/include/storage/dsm.h | 2 +- 3 files changed, 39 insertions(+), 25 deletions(-) diff --git a/contrib/test_shm_mq/setup.c b/contrib/test_shm_mq/setup.c index 572cf88..897c47b 100644 --- a/contrib/test_shm_mq/setup.c +++ b/contrib/test_shm_mq/setup.c @@ -125,7 +125,7 @@ setup_dynamic_shared_memory(int64 queue_size, int nworkers, segsize = shm_toc_estimate(&e); /* Create the shared memory segment and establish a table of contents. */ - seg = dsm_create(shm_toc_estimate(&e)); + seg = dsm_create("test_shm_mq", shm_toc_estimate(&e)); toc = shm_toc_create(PG_TEST_SHM_MQ_MAGIC, dsm_segment_address(seg), segsize); diff --git a/src/backend/storage/ipc/dsm.c b/src/backend/storage/ipc/dsm.c index a5c0084..66e24f0 100644 --- a/src/backend/storage/ipc/dsm.c +++ b/src/backend/storage/ipc/dsm.c @@ -80,8 +80,10 @@ struct dsm_segment /* Shared-memory state for a dynamic shared memory segment. */ typedef struct dsm_control_item { - dsm_handle handle; + dsm_handle handle; /* segment identifier */ uint32 refcnt; /* 2+ = active, 1 = moribund, 0 = gone */ + Size size; /* current size */ + char name[SHMEM_INDEX_KEYSIZE]; /* informational name */ } dsm_control_item; /* Layout of the dynamic shared memory control segment. */ @@ -454,14 +456,16 @@ dsm_set_control_handle(dsm_handle h) * Create a new dynamic shared memory segment. */ dsm_segment * -dsm_create(Size size) +dsm_create(const char *name, Size size) { dsm_segment *seg = dsm_create_descriptor(); - uint32 i; - uint32 nitems; + dsm_control_item *item; + uint32 slot; /* Unsafe in postmaster (and pointless in a stand-alone backend). */ Assert(IsUnderPostmaster); + Assert(name != NULL && strlen(name) > 0 && + strlen(name) < SHMEM_INDEX_KEYSIZE); if (!dsm_init_done) dsm_backend_startup(); @@ -479,33 +483,39 @@ dsm_create(Size size) /* Lock the control segment so we can register the new segment. */ LWLockAcquire(DynamicSharedMemoryControlLock, LW_EXCLUSIVE); - /* Search the control segment for an unused slot. */ - nitems = dsm_control->nitems; - for (i = 0; i < nitems; ++i) + /* + * Search the control segment for an unused slot that's previously been + * used. If we don't find one initialize a new one if there's still space. + */ + for (slot = 0; slot < dsm_control->nitems; ++slot) { - if (dsm_control->item[i].refcnt == 0) - { - dsm_control->item[i].handle = seg->handle; - /* refcnt of 1 triggers destruction, so start at 2 */ - dsm_control->item[i].refcnt = 2; - seg->control_slot = i; - LWLockRelease(DynamicSharedMemoryControlLock); - return seg; - } + if (dsm_control->item[slot].refcnt == 0) + break; } - /* Verify that we can support an additional mapping. */ - if (nitems >= dsm_control->maxitems) + /* Verify that we can support the mapping. */ + if (slot >= dsm_control->maxitems) ereport(ERROR, (errcode(ERRCODE_INSUFFICIENT_RESOURCES), errmsg("too many dynamic shared memory segments"))); - /* Enter the handle into a new array slot. */ - dsm_control->item[nitems].handle = seg->handle; + item = &dsm_control->item[slot]; + item->handle = seg->handle; /* refcnt of 1 triggers destruction, so start at 2 */ - dsm_control->item[nitems].refcnt = 2; - seg->control_slot = nitems; - dsm_control->nitems++; + item->refcnt = 2; + item->size = size; + strncpy(item->name, name, SHMEM_INDEX_SIZE - 1); + item->name[SHMEM_INDEX_SIZE] = 0; + + seg->control_slot = slot; + + /* + * Increase number of initilized slots if we didn't reuse a previously + * used one. + */ + if (slot >= dsm_control->nitems) + dsm_control->nitems++; + LWLockRelease(DynamicSharedMemoryControlLock); return seg; @@ -658,6 +668,10 @@ dsm_resize(dsm_segment *seg, Size size) Assert(seg->control_slot != INVALID_CONTROL_SLOT); dsm_impl_op(DSM_OP_RESIZE, seg->handle, size, &seg->impl_private, &seg->mapped_address, &seg->mapped_size, ERROR); + + /* persist the changed size */ + dsm_control->item[seg->control_slot].size = size; + return seg->mapped_address; } diff --git a/src/include/storage/dsm.h b/src/include/storage/dsm.h index 1d0110d..3dbe53b 100644 --- a/src/include/storage/dsm.h +++ b/src/include/storage/dsm.h @@ -29,7 +29,7 @@ extern void dsm_set_control_handle(dsm_handle h); #endif /* Functions that create, update, or remove mappings. */ -extern dsm_segment *dsm_create(Size size); +extern dsm_segment *dsm_create(const char *name, Size size); extern dsm_segment *dsm_attach(dsm_handle h); extern void *dsm_resize(dsm_segment *seg, Size size); extern void *dsm_remap(dsm_segment *seg); -- 1.8.5.rc2.dirty
>From 24c42133a4017a9b6caf60c0f824b8f8eaed9d51 Mon Sep 17 00:00:00 2001 From: Andres Freund <and...@anarazel.de> Date: Sun, 4 May 2014 13:37:20 +0200 Subject: [PATCH 2/2] Add views to see shared memory allocations. --- src/backend/catalog/system_views.sql | 6 ++ src/backend/storage/ipc/dsm.c | 80 ++++++++++++++++++++++ src/backend/storage/ipc/shmem.c | 124 +++++++++++++++++++++++++++++++++++ src/include/catalog/pg_proc.h | 4 ++ src/include/utils/builtins.h | 5 ++ src/test/regress/expected/rules.out | 9 +++ 6 files changed, 228 insertions(+) diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 42a4c00..c414260 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -387,6 +387,12 @@ CREATE VIEW pg_timezone_abbrevs AS CREATE VIEW pg_timezone_names AS SELECT * FROM pg_timezone_names(); +CREATE VIEW pg_static_shmem_allocations AS + SELECT * FROM pg_get_static_shmem_allocations(); + +CREATE VIEW pg_dynamic_shmem_allocations AS + SELECT * FROM pg_get_dynamic_shmem_allocations(); + -- Statistics views CREATE VIEW pg_stat_all_tables AS diff --git a/src/backend/storage/ipc/dsm.c b/src/backend/storage/ipc/dsm.c index 66e24f0..a8c6c38 100644 --- a/src/backend/storage/ipc/dsm.c +++ b/src/backend/storage/ipc/dsm.c @@ -34,12 +34,15 @@ #endif #include <sys/stat.h> +#include "fmgr.h" +#include "funcapi.h" #include "lib/ilist.h" #include "miscadmin.h" #include "storage/dsm.h" #include "storage/ipc.h" #include "storage/lwlock.h" #include "storage/pg_shmem.h" +#include "utils/builtins.h" #include "utils/guc.h" #include "utils/memutils.h" #include "utils/resowner_private.h" @@ -1022,3 +1025,80 @@ dsm_control_bytes_needed(uint32 nitems) return offsetof(dsm_control_header, item) +sizeof(dsm_control_item) * (uint64) nitems; } + +/* SQL SRF showing allocated shared memory */ +Datum +pg_get_dynamic_shmem_allocations(PG_FUNCTION_ARGS) +{ +#define PG_GET_SHMEM_SIZES_COLS 4 + + ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo; + TupleDesc tupdesc; + Tuplestorestate *tupstore; + MemoryContext per_query_ctx; + MemoryContext oldcontext; + int i; + + /* check to see if caller supports us returning a tuplestore */ + if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo)) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("set-valued function called in context that cannot accept a set"))); + if (!(rsinfo->allowedModes & SFRM_Materialize)) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("materialize mode required, but it is not " \ + "allowed in this context"))); + + /* Build a tuple descriptor for our result type */ + if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE) + elog(ERROR, "return type must be a row type"); + + per_query_ctx = rsinfo->econtext->ecxt_per_query_memory; + oldcontext = MemoryContextSwitchTo(per_query_ctx); + + tupstore = tuplestore_begin_heap(true, false, work_mem); + rsinfo->returnMode = SFRM_Materialize; + rsinfo->setResult = tupstore; + rsinfo->setDesc = tupdesc; + + MemoryContextSwitchTo(oldcontext); + + LWLockAcquire(DynamicSharedMemoryControlLock, LW_SHARED); + + for (i = 0; i < dsm_control->nitems; ++i) + { + dsm_control_item *item; + Datum values[PG_GET_SHMEM_SIZES_COLS]; + bool nulls[PG_GET_SHMEM_SIZES_COLS]; + + /* don't look at unused or about to be destroyed items */ + if (dsm_control->item[i].refcnt < 2) + continue; + + item = &dsm_control->item[i]; + + /* handle */ + values[0] = Int64GetDatum(item->handle); + nulls[0] = false; + + /* name */ + values[1] = CStringGetTextDatum(item->name); + nulls[1] = false; + + /* size */ + values[2] = Int64GetDatum(item->size); + nulls[2] = false; + + /* refcnt */ + values[3] = Int64GetDatum(item->refcnt - 2); + nulls[3] = false; + + tuplestore_putvalues(tupstore, tupdesc, values, nulls); + } + LWLockRelease(DynamicSharedMemoryControlLock); + + tuplestore_donestoring(tupstore); + + return (Datum) 0; +} diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c index 2ea2216..9f08c4e 100644 --- a/src/backend/storage/ipc/shmem.c +++ b/src/backend/storage/ipc/shmem.c @@ -66,11 +66,14 @@ #include "postgres.h" #include "access/transam.h" +#include "fmgr.h" +#include "funcapi.h" #include "miscadmin.h" #include "storage/lwlock.h" #include "storage/pg_shmem.h" #include "storage/shmem.h" #include "storage/spin.h" +#include "utils/builtins.h" /* shared memory global variables */ @@ -459,3 +462,124 @@ mul_size(Size s1, Size s2) errmsg("requested shared memory size overflows size_t"))); return result; } + +/* SQL SRF showing allocated shared memory */ +Datum +pg_get_static_shmem_allocations(PG_FUNCTION_ARGS) +{ +#define PG_GET_SHMEM_SIZES_COLS 4 + ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo; + TupleDesc tupdesc; + Tuplestorestate *tupstore; + MemoryContext per_query_ctx; + MemoryContext oldcontext; + HASH_SEQ_STATUS hstat; + ShmemIndexEnt *ent; + Size named_allocated = 0; + + /* check to see if caller supports us returning a tuplestore */ + if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo)) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("set-valued function called in context that cannot accept a set"))); + if (!(rsinfo->allowedModes & SFRM_Materialize)) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("materialize mode required, but it is not " \ + "allowed in this context"))); + + /* Build a tuple descriptor for our result type */ + if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE) + elog(ERROR, "return type must be a row type"); + + per_query_ctx = rsinfo->econtext->ecxt_per_query_memory; + oldcontext = MemoryContextSwitchTo(per_query_ctx); + + tupstore = tuplestore_begin_heap(true, false, work_mem); + rsinfo->returnMode = SFRM_Materialize; + rsinfo->setResult = tupstore; + rsinfo->setDesc = tupdesc; + + MemoryContextSwitchTo(oldcontext); + + hash_seq_init(&hstat, ShmemIndex); + + LWLockAcquire(ShmemIndexLock, LW_SHARED); + + /* output all allocated entries */ + while ((ent = (ShmemIndexEnt *) hash_seq_search(&hstat)) != NULL) + { + Datum values[PG_GET_SHMEM_SIZES_COLS]; + bool nulls[PG_GET_SHMEM_SIZES_COLS]; + + /* key */ + values[0] = CStringGetTextDatum(ent->key); + nulls[0] = false; + + /* off */ + values[1] = Int64GetDatum((char *) ent->location - (char *) ShmemSegHdr); + nulls[1] = false; + + /* size */ + values[2] = Int64GetDatum(ent->size); + nulls[2] = false; + named_allocated += ent->size; + + /* allocated */ + values[3] = BoolGetDatum(true); + nulls[3] = false; + + tuplestore_putvalues(tupstore, tupdesc, values, nulls); + } + + /* output shared memory allocated but not counted via the shmem index */ + { + Datum values[PG_GET_SHMEM_SIZES_COLS]; + bool nulls[PG_GET_SHMEM_SIZES_COLS]; + + /* key, show unnamed as NULL */ + nulls[0] = true; + + /* off */ + nulls[1] = true; + + /* size */ + values[2] = Int64GetDatum(ShmemSegHdr->freeoffset - named_allocated); + nulls[2] = false; + + /* allocated */ + values[3] = BoolGetDatum(true); + nulls[3] = false; + + tuplestore_putvalues(tupstore, tupdesc, values, nulls); + } + + /* output as-of-yet unused shared memory */ + { + Datum values[PG_GET_SHMEM_SIZES_COLS]; + bool nulls[PG_GET_SHMEM_SIZES_COLS]; + + /* key, show unallocated as NULL */ + nulls[0] = true; + + /* off */ + values[1] = Int64GetDatum(ShmemSegHdr->freeoffset); + nulls[1] = false; + + /* size */ + values[2] = Int64GetDatum(ShmemSegHdr->totalsize - ShmemSegHdr->freeoffset); + nulls[2] = false; + + /* allocated */ + values[3] = BoolGetDatum(false); + nulls[3] = false; + + tuplestore_putvalues(tupstore, tupdesc, values, nulls); + } + + LWLockRelease(ShmemIndexLock); + + tuplestore_donestoring(tupstore); + + return (Datum) 0; +} diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h index e601ccd..b59cfee 100644 --- a/src/include/catalog/pg_proc.h +++ b/src/include/catalog/pg_proc.h @@ -3899,6 +3899,10 @@ DATA(insert OID = 3035 ( pg_listening_channels PGNSP PGUID 12 1 10 0 0 f f f f DESCR("get the channels that the current backend listens to"); DATA(insert OID = 3036 ( pg_notify PGNSP PGUID 12 1 0 0 0 f f f f f f v 2 0 2278 "25 25" _null_ _null_ _null_ _null_ pg_notify _null_ _null_ _null_ )); DESCR("send a notification event"); +DATA(insert OID = 86 ( pg_get_static_shmem_allocations PGNSP PGUID 12 1 10 0 0 f f f f f t s 0 0 2249 "" "{25,20,20,16}" "{o,o,o,o}" "{key, off, size, allocated}" _null_ pg_get_static_shmem_allocations _null_ _null_ _null_ )); +DESCR("show static shared memory allocations"); +DATA(insert OID = 87 ( pg_get_dynamic_shmem_allocations PGNSP PGUID 12 1 10 0 0 f f f f f t s 0 0 2249 "" "{20,25,20,20}" "{o,o,o,o}" "{handle, name, size, refcnt}" _null_ pg_get_dynamic_shmem_allocations _null_ _null_ _null_ )); +DESCR("show dynamic shared memory allocations"); /* non-persistent series generator */ DATA(insert OID = 1066 ( generate_series PGNSP PGUID 12 1 1000 0 0 f f f f t t i 3 0 23 "23 23 23" _null_ _null_ _null_ _null_ generate_series_step_int4 _null_ _null_ _null_ )); diff --git a/src/include/utils/builtins.h b/src/include/utils/builtins.h index bbb5d39..ea91014 100644 --- a/src/include/utils/builtins.h +++ b/src/include/utils/builtins.h @@ -1209,4 +1209,9 @@ extern Datum pg_prepared_statement(PG_FUNCTION_ARGS); /* utils/mmgr/portalmem.c */ extern Datum pg_cursor(PG_FUNCTION_ARGS); +/* backend/storage/ipc/shmem.c */ +extern Datum pg_get_static_shmem_allocations(PG_FUNCTION_ARGS); +/* backend/storage/ipc/dsm.c */ +extern Datum pg_get_dynamic_shmem_allocations(PG_FUNCTION_ARGS); + #endif /* BUILTINS_H */ diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 87870cf..66937d5 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1308,6 +1308,10 @@ pg_cursors| SELECT c.name, c.is_scrollable, c.creation_time FROM pg_cursor() c(name, statement, is_holdable, is_binary, is_scrollable, creation_time); +pg_dynamic_shmem_allocations| SELECT pg_get_dynamic_shmem_allocations.key, + pg_get_dynamic_shmem_allocations.size, + pg_get_dynamic_shmem_allocations.refcnt + FROM pg_get_dynamic_shmem_allocations() pg_get_dynamic_shmem_allocations(key, size, refcnt); pg_group| SELECT pg_authid.rolname AS groname, pg_authid.oid AS grosysid, ARRAY( SELECT pg_auth_members.member @@ -1848,6 +1852,11 @@ pg_stat_xact_user_tables| SELECT pg_stat_xact_all_tables.relid, pg_stat_xact_all_tables.n_tup_hot_upd FROM pg_stat_xact_all_tables WHERE ((pg_stat_xact_all_tables.schemaname <> ALL (ARRAY['pg_catalog'::name, 'information_schema'::name])) AND (pg_stat_xact_all_tables.schemaname !~ '^pg_toast'::text)); +pg_static_shmem_allocations| SELECT pg_get_static_shmem_allocations.key, + pg_get_static_shmem_allocations.off, + pg_get_static_shmem_allocations.size, + pg_get_static_shmem_allocations.allocated + FROM pg_get_static_shmem_allocations() pg_get_static_shmem_allocations(key, off, size, allocated); pg_statio_all_indexes| SELECT c.oid AS relid, i.oid AS indexrelid, n.nspname AS schemaname, -- 1.8.5.rc2.dirty
-- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers