On Tue, Mar 17, 2020 at 02:04:01PM -0500, Justin Pryzby wrote: > > The example in the documentation could be better indented. Also, ISTM that > > there are two implicit laterals (format & pg_ls_dir_recurse) that I would > > make explicit. I'd use the pcs alias explicitely. I'd use meaningful aliases > > (eg ts instead of b, …). > > > On reflection, I think that a boolean "isdir" column is a bad idea because > > it is not extensible. I'd propose to switch to the standard "ls" approach of > > providing the type as one character: '-' for regular, 'd' for directory, 'l' > > for link, 's' for socket, 'c' for character special… > > I think that's outside the scope of the patch, since I'd want to change > pg_stat_file; that's where I borrowed "isdir" from, for consistency. > > Note that both LS_DIR_HISTORIC and LS_DIR_MODERN include LS_DIR_SKIP_SPECIAL, > so only pg_ls_dir itself show specials, so they way to do it would be to 1) > change pg_stat_file to expose the file's "type", 2) use pg_ls_dir() AS a, > lateral pg_stat_file(a) AS b, 3) then consider also changing LS_DIR_MODERN and > all the existing pg_ls_*.
The patch intends to fix the issue of "failing to show failed filesets" (because dirs are skipped) while also generalizing existing functions (to show directories and "isdir" column) and providing some more flexible ones (to list file and metadata of a dir, which is currently possible [only] for "special" directories, or by recursively calling pg_stat_file). I'm still of the opinion that supporting arbitrary file types is out of scope, but I changed the "isdir" to show "type". I'm only supporting '[-dl]'. I don't want to have to check #ifdef S_ISDOOR or whatever other vendors have. I insist that it is a separate patch, since it depends on everything else, and I have no feedback from anybody else as to whether any of that is desired. template1=# SELECT * FROM pg_ls_waldir(); name | size | access | modification | change | creation | type --------------------------+----------+------------------------+------------------------+------------------------+----------+------ barr | 0 | 2020-03-31 14:43:11-05 | 2020-03-31 14:43:11-05 | 2020-03-31 14:43:11-05 | | ? baz | 4096 | 2020-03-31 14:39:18-05 | 2020-03-31 14:39:18-05 | 2020-03-31 14:39:18-05 | | d foo | 0 | 2020-03-31 14:39:37-05 | 2020-03-31 14:39:37-05 | 2020-03-31 14:39:37-05 | | - archive_status | 4096 | 2020-03-31 14:38:20-05 | 2020-03-31 14:38:18-05 | 2020-03-31 14:38:18-05 | | d 000000010000000000000001 | 16777216 | 2020-03-31 14:42:53-05 | 2020-03-31 14:43:08-05 | 2020-03-31 14:43:08-05 | | - bar | 3 | 2020-03-31 14:39:16-05 | 2020-03-31 14:39:01-05 | 2020-03-31 14:39:01-05 | | l
>From f6b54482cc1d1b9595f49211b14807a20f994a3f Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Mon, 16 Mar 2020 14:12:55 -0500 Subject: [PATCH v15 01/10] Document historic behavior of links to directories.. Backpatch to 9.5: pg_stat_file --- doc/src/sgml/func.sgml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 7a0bb0c70a..25b1278459 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21558,7 +21558,8 @@ SELECT convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8'); size, last accessed time stamp, last modified time stamp, last file status change time stamp (Unix platforms only), file creation time stamp (Windows only), and a <type>boolean</type> - indicating if it is a directory. Typical usages include: + indicating if it is a directory (or a symbolic link to a directory). + Typical usages include: <programlisting> SELECT * FROM pg_stat_file('filename'); SELECT (pg_stat_file('filename')).modification; -- 2.17.0
>From e1c8750fb777df4bf41f36df7ae6cde39b907ec2 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Mon, 30 Mar 2020 18:59:16 -0500 Subject: [PATCH v15 02/10] pg_stat_file and pg_ls_dir_* to use lstat().. pg_ls_dir_* will now skip (no longer show) symbolic links, same as other non-regular file types, as we advertize we do since 8b6d94cf6. That seems to be the intented behavior, since irregular file types are 1) less portable; and, 2) we don't currently show a file's type except for "bool is_dir". pg_stat_file will now 1) show metadata of links themselves, rather than their target; and, 2) specifically, show links to directories with "is_dir=false"; and, 3) not error on broken symlinks. --- doc/src/sgml/func.sgml | 2 +- src/backend/utils/adt/genfile.c | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 25b1278459..630f03d7dd 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21558,7 +21558,7 @@ SELECT convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8'); size, last accessed time stamp, last modified time stamp, last file status change time stamp (Unix platforms only), file creation time stamp (Windows only), and a <type>boolean</type> - indicating if it is a directory (or a symbolic link to a directory). + indicating if it is a directory. Typical usages include: <programlisting> SELECT * FROM pg_stat_file('filename'); diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c index ceaa6180da..219ac160f8 100644 --- a/src/backend/utils/adt/genfile.c +++ b/src/backend/utils/adt/genfile.c @@ -370,7 +370,7 @@ pg_stat_file(PG_FUNCTION_ARGS) filename = convert_and_check_filename(filename_t); - if (stat(filename, &fst) < 0) + if (lstat(filename, &fst) < 0) { if (missing_ok && errno == ENOENT) PG_RETURN_NULL(); @@ -596,7 +596,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok) /* Get the file info */ snprintf(path, sizeof(path), "%s/%s", dir, de->d_name); - if (stat(path, &attrib) < 0) + if (lstat(path, &attrib) < 0) { /* Ignore concurrently-deleted files, else complain */ if (errno == ENOENT) -- 2.17.0
>From 4c6e234cfee829f0e14f1d4cfb6c0a7c77c86c9b Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Tue, 17 Mar 2020 13:16:24 -0500 Subject: [PATCH v15 03/10] Add tests on pg_ls_dir before changing it --- src/test/regress/expected/misc_functions.out | 18 ++++++++++++++++++ src/test/regress/sql/misc_functions.sql | 5 +++++ 2 files changed, 23 insertions(+) diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out index d3acb98d04..2e87c548eb 100644 --- a/src/test/regress/expected/misc_functions.out +++ b/src/test/regress/expected/misc_functions.out @@ -201,6 +201,24 @@ select count(*) > 0 from t (1 row) +select * from (select pg_ls_dir('.', false, true) as name) as ls where ls.name='.'; -- include_dot_dirs=true + name +------ + . +(1 row) + +select * from (select pg_ls_dir('.', false, false) as name) as ls where ls.name='.'; -- include_dot_dirs=false + name +------ +(0 rows) + +select pg_ls_dir('does not exist', true, false); -- ok with missingok=true + pg_ls_dir +----------- +(0 rows) + +select pg_ls_dir('does not exist'); -- fails with missingok=false +ERROR: could not open directory "does not exist": No such file or directory -- -- Test adding a support function to a subject function -- diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql index 094e8f8296..f6857ad177 100644 --- a/src/test/regress/sql/misc_functions.sql +++ b/src/test/regress/sql/misc_functions.sql @@ -60,6 +60,11 @@ select count(*) > 0 from where spcname = 'pg_default') pts join pg_database db on pts.pts = db.oid; +select * from (select pg_ls_dir('.', false, true) as name) as ls where ls.name='.'; -- include_dot_dirs=true +select * from (select pg_ls_dir('.', false, false) as name) as ls where ls.name='.'; -- include_dot_dirs=false +select pg_ls_dir('does not exist', true, false); -- ok with missingok=true +select pg_ls_dir('does not exist'); -- fails with missingok=false + -- -- Test adding a support function to a subject function -- -- 2.17.0
>From e255804b9e50e869bb2ba7538349ff2b9cb26133 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Mon, 9 Mar 2020 22:40:24 -0500 Subject: [PATCH v15 04/10] Add pg_ls_dir_metadata to list a dir with file metadata.. Generalize pg_ls_dir_files and retire pg_ls_dir XXX: Change to use lstat() to allow pg_ls_dir_recurse to avoid infinite recursion. That means: - links to dirs are NOT SHOWN (was: shown with isdir=false;) - NO: timestamps shown are those of the link; - changed pg_stat_file for consistency; Need catversion bumped? --- doc/src/sgml/func.sgml | 17 ++ src/backend/catalog/system_views.sql | 1 + src/backend/utils/adt/genfile.c | 227 +++++++++++-------- src/include/catalog/pg_proc.dat | 12 + src/test/regress/expected/misc_functions.out | 24 ++ src/test/regress/input/tablespace.source | 5 + src/test/regress/output/tablespace.source | 8 + src/test/regress/sql/misc_functions.sql | 11 + 8 files changed, 214 insertions(+), 91 deletions(-) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 630f03d7dd..ed8e1977ba 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21372,6 +21372,15 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); List the contents of a directory. Restricted to superusers by default, but other users can be granted EXECUTE to run the function. </entry> </row> + <row> + <entry> + <literal><function>pg_ls_dir_metadata(<parameter>dirname</parameter> <type>text</type> [, <parameter>missing_ok</parameter> <type>boolean</type>, <parameter>include_dot_dirs</parameter> <type>boolean</type>])</function></literal> + </entry> + <entry><type>setof text</type></entry> + <entry> + For each file in a directory, list the file and its metadata. Restricted to superusers by default, but other users can be granted EXECUTE to run the function. + </entry> + </row> <row> <entry> <literal><function>pg_ls_logdir()</function></literal> @@ -21472,6 +21481,14 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); empty directory from an non-existent directory. </para> + <indexterm> + <primary>pg_ls_dir_metadata</primary> + </indexterm> + <para> + <function>pg_ls_dir_metadata</function> lists the files in the specified + directory along with the file's metadata. + </para> + <indexterm> <primary>pg_ls_logdir</primary> </indexterm> diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 83d00c6cde..4511f9d658 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1436,6 +1436,7 @@ REVOKE EXECUTE ON FUNCTION pg_stat_file(text,boolean) FROM public; REVOKE EXECUTE ON FUNCTION pg_ls_dir(text) FROM public; REVOKE EXECUTE ON FUNCTION pg_ls_dir(text,boolean,boolean) FROM public; +REVOKE EXECUTE ON FUNCTION pg_ls_dir_metadata(text,boolean,boolean) FROM public; -- -- We also set up some things as accessible to standard roles. diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c index 219ac160f8..eaa7c548e0 100644 --- a/src/backend/utils/adt/genfile.c +++ b/src/backend/utils/adt/genfile.c @@ -36,6 +36,21 @@ #include "utils/syscache.h" #include "utils/timestamp.h" +static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags); + +#define LS_DIR_ISDIR (1<<0) /* Show column: isdir */ +#define LS_DIR_METADATA (1<<1) /* Show columns: mtime, size */ +#define LS_DIR_MISSING_OK (1<<2) /* Ignore ENOENT if the toplevel dir is missing */ +#define LS_DIR_SKIP_DOT_DIRS (1<<3) /* Do not show . or .. */ +#define LS_DIR_SKIP_HIDDEN (1<<4) /* Do not show anything begining with . */ +#define LS_DIR_SKIP_DIRS (1<<5) /* Do not show directories */ +#define LS_DIR_SKIP_SPECIAL (1<<6) /* Do not show special file types */ + +/* + * Shortcut for the historic behavior of the pg_ls_* functions (not including + * pg_ls_dir, which skips different files and doesn't show metadata. + */ +#define LS_DIR_HISTORIC (LS_DIR_SKIP_DIRS|LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA) /* * Convert a "text" filename argument to C string, and check it's allowable. @@ -413,6 +428,11 @@ pg_stat_file(PG_FUNCTION_ARGS) values[4] = TimestampTzGetDatum(time_t_to_timestamptz(fst.st_ctime)); #endif values[5] = BoolGetDatum(S_ISDIR(fst.st_mode)); +#ifdef WIN32 + /* Links should have isdir=false */ + if (pgwin32_is_junction(filename)) + values[5] = BoolGetDatum(false); +#endif tuple = heap_form_tuple(tupdesc, values, isnull); @@ -440,79 +460,9 @@ pg_stat_file_1arg(PG_FUNCTION_ARGS) Datum pg_ls_dir(PG_FUNCTION_ARGS) { - ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo; - char *location; - bool missing_ok = false; - bool include_dot_dirs = false; - bool randomAccess; - TupleDesc tupdesc; - Tuplestorestate *tupstore; - DIR *dirdesc; - struct dirent *de; - MemoryContext oldcontext; - - location = convert_and_check_filename(PG_GETARG_TEXT_PP(0)); - - /* check the optional arguments */ - if (PG_NARGS() == 3) - { - if (!PG_ARGISNULL(1)) - missing_ok = PG_GETARG_BOOL(1); - if (!PG_ARGISNULL(2)) - include_dot_dirs = PG_GETARG_BOOL(2); - } - - /* check to see if caller supports us returning a tuplestore */ - if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo)) - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("set-valued function called in context that cannot accept a set"))); - if (!(rsinfo->allowedModes & SFRM_Materialize)) - ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("materialize mode required, but it is not allowed in this context"))); - - /* The tupdesc and tuplestore must be created in ecxt_per_query_memory */ - oldcontext = MemoryContextSwitchTo(rsinfo->econtext->ecxt_per_query_memory); - - tupdesc = CreateTemplateTupleDesc(1); - TupleDescInitEntry(tupdesc, (AttrNumber) 1, "pg_ls_dir", TEXTOID, -1, 0); - - randomAccess = (rsinfo->allowedModes & SFRM_Materialize_Random) != 0; - tupstore = tuplestore_begin_heap(randomAccess, false, work_mem); - rsinfo->returnMode = SFRM_Materialize; - rsinfo->setResult = tupstore; - rsinfo->setDesc = tupdesc; - - MemoryContextSwitchTo(oldcontext); - - dirdesc = AllocateDir(location); - if (!dirdesc) - { - /* Return empty tuplestore if appropriate */ - if (missing_ok && errno == ENOENT) - return (Datum) 0; - /* Otherwise, we can let ReadDir() throw the error */ - } - - while ((de = ReadDir(dirdesc, location)) != NULL) - { - Datum values[1]; - bool nulls[1]; - - if (!include_dot_dirs && - (strcmp(de->d_name, ".") == 0 || - strcmp(de->d_name, "..") == 0)) - continue; - - values[0] = CStringGetTextDatum(de->d_name); - nulls[0] = false; - - tuplestore_putvalues(tupstore, tupdesc, values, nulls); - } - - FreeDir(dirdesc); - return (Datum) 0; + text *filename_t = PG_GETARG_TEXT_PP(0); + char *filename = convert_and_check_filename(filename_t); + return pg_ls_dir_files(fcinfo, filename, LS_DIR_SKIP_DOT_DIRS); } /* @@ -525,7 +475,9 @@ pg_ls_dir(PG_FUNCTION_ARGS) Datum pg_ls_dir_1arg(PG_FUNCTION_ARGS) { - return pg_ls_dir(fcinfo); + text *filename_t = PG_GETARG_TEXT_PP(0); + char *filename = convert_and_check_filename(filename_t); + return pg_ls_dir_files(fcinfo, filename, LS_DIR_SKIP_DOT_DIRS); } /* @@ -535,7 +487,7 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS) * Other unreadable-directory cases throw an error. */ static Datum -pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok) +pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags) { ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo; bool randomAccess; @@ -544,6 +496,32 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok) DIR *dirdesc; struct dirent *de; MemoryContext oldcontext; + TypeFuncClass tuptype ; + + /* isdir depends on metadata */ + Assert(!(flags&LS_DIR_ISDIR) || (flags&LS_DIR_METADATA)); + /* Unreasonable to show isdir and skip dirs */ + Assert(!(flags&LS_DIR_ISDIR) || !(flags&LS_DIR_SKIP_DIRS)); + + /* check the optional arguments */ + if (PG_NARGS() == 3) + { + if (!PG_ARGISNULL(1)) + { + if (PG_GETARG_BOOL(1)) + flags |= LS_DIR_MISSING_OK; + else + flags &= ~LS_DIR_MISSING_OK; + } + + if (!PG_ARGISNULL(2)) + { + if (PG_GETARG_BOOL(2)) + flags &= ~LS_DIR_SKIP_DOT_DIRS; + else + flags |= LS_DIR_SKIP_DOT_DIRS; + } + } /* check to see if caller supports us returning a tuplestore */ if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo)) @@ -558,8 +536,18 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok) /* The tupdesc and tuplestore must be created in ecxt_per_query_memory */ oldcontext = MemoryContextSwitchTo(rsinfo->econtext->ecxt_per_query_memory); - if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE) - elog(ERROR, "return type must be a row type"); + tuptype = get_call_result_type(fcinfo, NULL, &tupdesc); + if (flags & LS_DIR_METADATA) + { + if (tuptype != TYPEFUNC_COMPOSITE) + elog(ERROR, "return type must be a row type"); + } else { + /* pg_ls_dir returns a simple scalar */ + if (tuptype != TYPEFUNC_SCALAR) + elog(ERROR, "return type must be a scalar type"); + tupdesc = CreateTemplateTupleDesc(1); + TupleDescInitEntry(tupdesc, (AttrNumber) 1, "column", TEXTOID, -1, 0); + } randomAccess = (rsinfo->allowedModes & SFRM_Materialize_Random) != 0; tupstore = tuplestore_begin_heap(randomAccess, false, work_mem); @@ -578,20 +566,27 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok) if (!dirdesc) { /* Return empty tuplestore if appropriate */ - if (missing_ok && errno == ENOENT) + if (flags&LS_DIR_MISSING_OK && errno == ENOENT) return (Datum) 0; /* Otherwise, we can let ReadDir() throw the error */ } while ((de = ReadDir(dirdesc, dir)) != NULL) { - Datum values[3]; - bool nulls[3]; + Datum values[4]; + bool nulls[4]; char path[MAXPGPATH * 2]; struct stat attrib; - /* Skip hidden files */ - if (de->d_name[0] == '.') + /* Skip dot dirs? */ + if (flags & LS_DIR_SKIP_DOT_DIRS && + (strcmp(de->d_name, ".") == 0 || + strcmp(de->d_name, "..") == 0)) + continue; + + /* Skip hidden files? */ + if (flags & LS_DIR_SKIP_HIDDEN && + de->d_name[0] == '.') continue; /* Get the file info */ @@ -606,13 +601,34 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok) errmsg("could not stat file \"%s\": %m", path))); } - /* Ignore anything but regular files */ - if (!S_ISREG(attrib.st_mode)) - continue; + /* Skip dirs or special files? */ + if (S_ISDIR(attrib.st_mode)) + { + if (flags & LS_DIR_SKIP_DIRS) + continue; + } + else if (!S_ISREG(attrib.st_mode)) + { + if (flags & LS_DIR_SKIP_SPECIAL) + continue; + } values[0] = CStringGetTextDatum(de->d_name); - values[1] = Int64GetDatum((int64) attrib.st_size); - values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime)); + if (flags & LS_DIR_METADATA) + { + values[1] = Int64GetDatum((int64) attrib.st_size); + values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime)); + if (flags & LS_DIR_ISDIR) + { + values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode)); +#ifdef WIN32 + /* Links should have isdir=false */ + if (pgwin32_is_junction(path)) + values[3] = BoolGetDatum(false); +#endif + } + } + memset(nulls, 0, sizeof(nulls)); tuplestore_putvalues(tupstore, tupdesc, values, nulls); @@ -626,14 +642,14 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok) Datum pg_ls_logdir(PG_FUNCTION_ARGS) { - return pg_ls_dir_files(fcinfo, Log_directory, false); + return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_HISTORIC); } /* Function to return the list of files in the WAL directory */ Datum pg_ls_waldir(PG_FUNCTION_ARGS) { - return pg_ls_dir_files(fcinfo, XLOGDIR, false); + return pg_ls_dir_files(fcinfo, XLOGDIR, LS_DIR_HISTORIC); } /* @@ -651,7 +667,8 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc) tblspc))); TempTablespacePath(path, tblspc); - return pg_ls_dir_files(fcinfo, path, true); + return pg_ls_dir_files(fcinfo, path, + LS_DIR_HISTORIC | LS_DIR_MISSING_OK); } /* @@ -680,5 +697,33 @@ pg_ls_tmpdir_1arg(PG_FUNCTION_ARGS) Datum pg_ls_archive_statusdir(PG_FUNCTION_ARGS) { - return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true); + return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", + LS_DIR_HISTORIC | LS_DIR_MISSING_OK); +} + +/* + * Function to return the list of files and metadata in an arbitrary directory. + */ +Datum +pg_ls_dir_metadata(PG_FUNCTION_ARGS) +{ + char *dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0)); + + return pg_ls_dir_files(fcinfo, dirname, + LS_DIR_METADATA | LS_DIR_SKIP_SPECIAL | LS_DIR_ISDIR); +} + +/* + * Function to return the list of files and metadata in an arbitrary directory. + * note: this wrapper is necessary to pass the sanity check in opr_sanity, + * which checks that all built-in functions that share the implementing C + * function take the same number of arguments. + */ +Datum +pg_ls_dir_metadata_1arg(PG_FUNCTION_ARGS) +{ + char *dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0)); + + return pg_ls_dir_files(fcinfo, dirname, + LS_DIR_METADATA | LS_DIR_SKIP_SPECIAL | LS_DIR_ISDIR); } diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index a6a708cca9..e8846e1522 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -10801,6 +10801,18 @@ proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}', proargnames => '{tablespace,name,size,modification}', prosrc => 'pg_ls_tmpdir_1arg' }, +{ oid => '5032', descr => 'list directory with metadata', + proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't', + provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool', + proallargtypes => '{text,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o}', + proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,modification,isdir}', + prosrc => 'pg_ls_dir_metadata' }, +{ oid => '5033', descr => 'list directory with metadata', + proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't', + provolatile => 'v', prorettype => 'record', proargtypes => 'text', + proallargtypes => '{text,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}', + proargnames => '{dirname,name,size,modification,isdir}', + prosrc => 'pg_ls_dir_metadata_1arg' }, # hash partitioning constraint function { oid => '5028', descr => 'hash partition CHECK constraint', diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out index 2e87c548eb..7930909f02 100644 --- a/src/test/regress/expected/misc_functions.out +++ b/src/test/regress/expected/misc_functions.out @@ -219,6 +219,30 @@ select pg_ls_dir('does not exist', true, false); -- ok with missingok=true select pg_ls_dir('does not exist'); -- fails with missingok=false ERROR: could not open directory "does not exist": No such file or directory +-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet +-- The name='' condition is never true, so the function runs to completion but returns zero rows. +select * from pg_ls_tmpdir() where name='Does not exist'; + name | size | modification +------+------+-------------- +(0 rows) + +select name, isdir from pg_ls_dir_metadata('.') where name='.'; + name | isdir +------+------- + . | t +(1 row) + +select name, isdir from pg_ls_dir_metadata('.', false, false) where name='.'; -- include_dot_dirs=false + name | isdir +------+------- +(0 rows) + +-- Check that expected columns are present +select * from pg_ls_dir_metadata('.') limit 0; + name | size | modification | isdir +------+------+--------------+------- +(0 rows) + -- -- Test adding a support function to a subject function -- diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/input/tablespace.source index a5f61a35dc..0b9cfe615e 100644 --- a/src/test/regress/input/tablespace.source +++ b/src/test/regress/input/tablespace.source @@ -11,6 +11,11 @@ DROP TABLESPACE regress_tblspacewith; -- create a tablespace we can use CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@'; +-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet +-- The name='' condition is never true, so the function runs to completion but returns zero rows. +-- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir() +SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist'; + -- try setting and resetting some properties for the new tablespace ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1); ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true); -- fail diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source index 162b591b31..a42714bf40 100644 --- a/src/test/regress/output/tablespace.source +++ b/src/test/regress/output/tablespace.source @@ -13,6 +13,14 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith'; DROP TABLESPACE regress_tblspacewith; -- create a tablespace we can use CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@'; +-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet +-- The name='' condition is never true, so the function runs to completion but returns zero rows. +-- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir() +SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist'; + name | size | modification +------+------+-------------- +(0 rows) + -- try setting and resetting some properties for the new tablespace ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1); ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true); -- fail diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql index f6857ad177..372345720d 100644 --- a/src/test/regress/sql/misc_functions.sql +++ b/src/test/regress/sql/misc_functions.sql @@ -65,6 +65,17 @@ select * from (select pg_ls_dir('.', false, false) as name) as ls where ls.name= select pg_ls_dir('does not exist', true, false); -- ok with missingok=true select pg_ls_dir('does not exist'); -- fails with missingok=false +-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet +-- The name='' condition is never true, so the function runs to completion but returns zero rows. +select * from pg_ls_tmpdir() where name='Does not exist'; + +select name, isdir from pg_ls_dir_metadata('.') where name='.'; + +select name, isdir from pg_ls_dir_metadata('.', false, false) where name='.'; -- include_dot_dirs=false + +-- Check that expected columns are present +select * from pg_ls_dir_metadata('.') limit 0; + -- -- Test adding a support function to a subject function -- -- 2.17.0
>From 3294321ae8a5a751b4e81e544e070a9106020b82 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Sun, 8 Mar 2020 22:57:54 -0500 Subject: [PATCH v15 05/10] pg_ls_tmpdir to show directories and "isdir" argument.. similar to pg_stat_file(). It's worth breaking the function's return type, since core postgres creates "shared filesets" underneath the temp dirs, and it's unreasonable to not show them here, and the alternative query to show them is unreasaonbly complicated. See following commit which also adds these columns to the other pg_ls_* functions. Although I don't think it matters that they're easily UNIONed, it'd still make great sense if they returned the same columns. Need catversion bump --- doc/src/sgml/func.sgml | 22 +++++++++++--------- src/backend/utils/adt/genfile.c | 4 ++-- src/include/catalog/pg_proc.dat | 8 +++---- src/test/regress/expected/misc_functions.out | 4 ++-- src/test/regress/output/tablespace.source | 4 ++-- 5 files changed, 22 insertions(+), 20 deletions(-) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index ed8e1977ba..343af5e954 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21421,12 +21421,12 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); </entry> <entry><type>setof record</type></entry> <entry> - List the name, size, and last modification time of files in the - temporary directory for <parameter>tablespace</parameter>. If - <parameter>tablespace</parameter> is not provided, the - <literal>pg_default</literal> tablespace is used. Access is granted - to members of the <literal>pg_monitor</literal> role and may be - granted to other non-superuser roles. + For the temporary directory within <parameter>tablespace</parameter>, + list each file's name, size, last modification time, and a boolean + indicating if it is a directory. If <parameter>tablespace</parameter> + is not provided, the <literal>pg_default</literal> tablespace is used. + Access is granted to members of the <literal>pg_monitor</literal> role + and may be granted to other non-superuser roles. </entry> </row> <row> @@ -21529,14 +21529,16 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); <primary>pg_ls_tmpdir</primary> </indexterm> <para> - <function>pg_ls_tmpdir</function> returns the name, size, and last modified - time (mtime) of each file in the temporary file directory for the specified - <parameter>tablespace</parameter>. If <parameter>tablespace</parameter> is + <function>pg_ls_tmpdir</function> lists each file in the temporary file + directory for the specified <parameter>tablespace</parameter>, along with + its size, last modified time (mtime) and a boolean indicating if the file is a + directory. Directories are used for temporary files shared by parallel + processes. If <parameter>tablespace</parameter> is not provided, the <literal>pg_default</literal> tablespace is used. By default only superusers and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. - Filenames beginning with a dot, directories, and other special files are not shown. + Filenames beginning with a dot and special file types are not shown. </para> <indexterm> diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c index eaa7c548e0..78edc85719 100644 --- a/src/backend/utils/adt/genfile.c +++ b/src/backend/utils/adt/genfile.c @@ -481,7 +481,7 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS) } /* - * Generic function to return a directory listing of files. + * Generic function to return a directory listing of files (and optionally dirs). * * If the directory isn't there, silently return an empty set if missing_ok. * Other unreadable-directory cases throw an error. @@ -668,7 +668,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc) TempTablespacePath(path, tblspc); return pg_ls_dir_files(fcinfo, path, - LS_DIR_HISTORIC | LS_DIR_MISSING_OK); + LS_DIR_SKIP_HIDDEN | LS_DIR_SKIP_SPECIAL | LS_DIR_ISDIR | LS_DIR_METADATA | LS_DIR_MISSING_OK); } /* diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index e8846e1522..40d951f029 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -10793,13 +10793,13 @@ { oid => '5029', descr => 'list files in the pgsql_tmp directory', proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}', - proargnames => '{name,size,modification}', prosrc => 'pg_ls_tmpdir_noargs' }, + proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}', + proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_noargs' }, { oid => '5030', descr => 'list files in the pgsql_tmp directory', proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => 'oid', - proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}', - proargnames => '{tablespace,name,size,modification}', + proallargtypes => '{oid,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}', + proargnames => '{tablespace,name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_1arg' }, { oid => '5032', descr => 'list directory with metadata', proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't', diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out index 7930909f02..38493de732 100644 --- a/src/test/regress/expected/misc_functions.out +++ b/src/test/regress/expected/misc_functions.out @@ -222,8 +222,8 @@ ERROR: could not open directory "does not exist": No such file or directory -- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet -- The name='' condition is never true, so the function runs to completion but returns zero rows. select * from pg_ls_tmpdir() where name='Does not exist'; - name | size | modification -------+------+-------------- + name | size | modification | isdir +------+------+--------------+------- (0 rows) select name, isdir from pg_ls_dir_metadata('.') where name='.'; diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source index a42714bf40..1c88e914e3 100644 --- a/src/test/regress/output/tablespace.source +++ b/src/test/regress/output/tablespace.source @@ -17,8 +17,8 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@'; -- The name='' condition is never true, so the function runs to completion but returns zero rows. -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir() SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist'; - name | size | modification -------+------+-------------- + name | size | modification | isdir +------+------+--------------+------- (0 rows) -- try setting and resetting some properties for the new tablespace -- 2.17.0
>From 374b0334ab7438c0155242fec26b81ae1ab3fb6b Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Mon, 9 Mar 2020 01:00:42 -0500 Subject: [PATCH v15 06/10] pg_ls_*dir to show directories and "isdir" column.. pg_ls_logdir, pg_ls_waldir, pg_ls_archive_statusdir Need catversion bump --- doc/src/sgml/func.sgml | 39 +++++++++++--------- src/backend/utils/adt/genfile.c | 11 ++++-- src/include/catalog/pg_proc.dat | 12 +++--- src/test/regress/expected/misc_functions.out | 4 +- 4 files changed, 37 insertions(+), 29 deletions(-) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 343af5e954..6a4623d59b 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21387,8 +21387,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); </entry> <entry><type>setof record</type></entry> <entry> - List the name, size, and last modification time of files in the log - directory. Access is granted to members of the <literal>pg_monitor</literal> + For each file in the log directory, list the file's name, size, last + modification time, and a boolean indicating if it is a directory. + Access is granted to members of the <literal>pg_monitor</literal> role and may be granted to other non-superuser roles. </entry> </row> @@ -21398,8 +21399,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); </entry> <entry><type>setof record</type></entry> <entry> - List the name, size, and last modification time of files in the WAL - directory. Access is granted to members of the <literal>pg_monitor</literal> + For each file in the WAL directory, list the file's name, size, last + modification time, and a boolean indicating if it is a directory. + Access is granted to members of the <literal>pg_monitor</literal> role and may be granted to other non-superuser roles. </entry> </row> @@ -21409,8 +21411,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); </entry> <entry><type>setof record</type></entry> <entry> - List the name, size, and last modification time of files in the WAL - archive status directory. Access is granted to members of the + For each file in the WAL archive status directory, list the file's + name, size, last modification time, and a boolean indicating if it is a + directory. Access is granted to members of the <literal>pg_monitor</literal> role and may be granted to other non-superuser roles. </entry> @@ -21493,36 +21496,38 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); <primary>pg_ls_logdir</primary> </indexterm> <para> - <function>pg_ls_logdir</function> returns the name, size, and last modified time - (mtime) of each file in the log directory. By default, only superusers + <function>pg_ls_logdir</function> lists each file in the log directory, + along with file's size, last modification time, and a boolean + indicating if the file is a directory. By default, only superusers and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. - Filenames beginning with a dot, directories, and other special files are not shown. + Filenames beginning with a dot and special file types are not shown. </para> <indexterm> <primary>pg_ls_waldir</primary> </indexterm> <para> - <function>pg_ls_waldir</function> returns the name, size, and last modified time - (mtime) of each file in the write ahead log (WAL) directory. By - default only superusers and members of the <literal>pg_monitor</literal> role + <function>pg_ls_waldir</function> lists each file in the WAL directory, + along with the file's size, last modification time, and a boolean + indicating if the file is a directory. By default, only superusers + and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. - Filenames beginning with a dot, directories, and other special files are not shown. + Filenames beginning with a dot and special file types are not shown. </para> <indexterm> <primary>pg_ls_archive_statusdir</primary> </indexterm> <para> - <function>pg_ls_archive_statusdir</function> returns the name, size, and - last modified time (mtime) of each file in the WAL archive status - directory <filename>pg_wal/archive_status</filename>. By default only + <function>pg_ls_archive_statusdir</function> lists each file in the WAL + archive status directory, along with the file's size, last modification + time, and a boolean indicating if the file is a directory. By default, only superusers and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. - Filenames beginning with a dot, directories, and other special files are not shown. + Filenames beginning with a dot and special file types are not shown. </para> <indexterm> diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c index 78edc85719..afef2bf4ab 100644 --- a/src/backend/utils/adt/genfile.c +++ b/src/backend/utils/adt/genfile.c @@ -52,6 +52,9 @@ static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags */ #define LS_DIR_HISTORIC (LS_DIR_SKIP_DIRS|LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA) +/* Shortcut for common behavior */ +#define LS_DIR_COMMON (LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA) + /* * Convert a "text" filename argument to C string, and check it's allowable. * @@ -642,14 +645,14 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags) Datum pg_ls_logdir(PG_FUNCTION_ARGS) { - return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_HISTORIC); + return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_COMMON); } /* Function to return the list of files in the WAL directory */ Datum pg_ls_waldir(PG_FUNCTION_ARGS) { - return pg_ls_dir_files(fcinfo, XLOGDIR, LS_DIR_HISTORIC); + return pg_ls_dir_files(fcinfo, XLOGDIR, LS_DIR_COMMON); } /* @@ -668,7 +671,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc) TempTablespacePath(path, tblspc); return pg_ls_dir_files(fcinfo, path, - LS_DIR_SKIP_HIDDEN | LS_DIR_SKIP_SPECIAL | LS_DIR_ISDIR | LS_DIR_METADATA | LS_DIR_MISSING_OK); + LS_DIR_COMMON | LS_DIR_MISSING_OK); } /* @@ -698,7 +701,7 @@ Datum pg_ls_archive_statusdir(PG_FUNCTION_ARGS) { return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", - LS_DIR_HISTORIC | LS_DIR_MISSING_OK); + LS_DIR_COMMON | LS_DIR_MISSING_OK); } /* diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 40d951f029..408bd40256 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -10777,18 +10777,18 @@ { oid => '3353', descr => 'list files in the log directory', proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}', - proargnames => '{name,size,modification}', prosrc => 'pg_ls_logdir' }, + proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}', + proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_logdir' }, { oid => '3354', descr => 'list of files in the WAL directory', proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}', - proargnames => '{name,size,modification}', prosrc => 'pg_ls_waldir' }, + proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}', + proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_waldir' }, { oid => '5031', descr => 'list of files in the archive_status directory', proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', - proargtypes => '', proallargtypes => '{text,int8,timestamptz}', - proargmodes => '{o,o,o}', proargnames => '{name,size,modification}', + proargtypes => '', proallargtypes => '{text,int8,timestamptz,bool}', + proargmodes => '{o,o,o,o}', proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_archive_statusdir' }, { oid => '5029', descr => 'list files in the pgsql_tmp directory', proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't', diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out index 38493de732..64b1417fb8 100644 --- a/src/test/regress/expected/misc_functions.out +++ b/src/test/regress/expected/misc_functions.out @@ -157,8 +157,8 @@ select count(*) > 0 as ok from (select pg_ls_waldir()) ss; -- Test not-run-to-completion cases. select * from pg_ls_waldir() limit 0; - name | size | modification -------+------+-------------- + name | size | modification | isdir +------+------+--------------+------- (0 rows) select count(*) > 0 as ok from (select * from pg_ls_waldir() limit 1) ss; -- 2.17.0
>From 9a9f031b65a85b3b4085db3c293321c7f1a8b56f Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Sun, 8 Mar 2020 22:52:14 -0500 Subject: [PATCH v15 07/10] Add pg_ls_dir_recurse to show dir recursively.. ..possibly there's a better place to put this, like maybe a doc-only example ? XXX: this test case is unstable, if backends/autovacuum remove FSM... ERROR: could not stat file "./base/16384/20345_fsm": No such file or directory CONTEXT: SQL function "pg_ls_dir_recurse" statement 1 Need catversion bumped ? --- doc/src/sgml/func.sgml | 25 ++++++++++++++++++++ src/backend/catalog/system_views.sql | 1 + src/include/catalog/pg_proc.dat | 6 +++++ src/test/regress/expected/misc_functions.out | 14 +++++++++++ src/test/regress/sql/misc_functions.sql | 6 +++++ 5 files changed, 52 insertions(+) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 6a4623d59b..7c8b102610 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21381,6 +21381,16 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); For each file in a directory, list the file and its metadata. Restricted to superusers by default, but other users can be granted EXECUTE to run the function. </entry> </row> + <row> + <entry> + <literal><function>pg_ls_dir_recurse(<parameter>dirname</parameter> <type>text</type>)</function></literal> + </entry> + <entry><type>setof text</type></entry> + <entry> + Call pg_ls_dir_metadata to recursively list the files in the specified directory, along with each file's metadata. + Restricted to superusers by default, but other users can be granted EXECUTE to run the function. + </entry> + </row> <row> <entry> <literal><function>pg_ls_logdir()</function></literal> @@ -21492,6 +21502,21 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); directory along with the file's metadata. </para> + <indexterm> + <primary>pg_ls_dir_recurse</primary> + </indexterm> + <para> + <function>pg_ls_dir_recurse</function> recursively lists the files + in the specified directory. + To recursively list temporary directories in all tablespaces: +<programlisting> +SELECT * FROM (SELECT DISTINCT COALESCE(NULLIF(pg_tablespace_location(b.oid),'')||suffix, 'base/pgsql_tmp') AS dir +FROM pg_tablespace b, pg_control_system() pcs, +lateral format('/PG_%s_%s', left(current_setting('server_version_num'), 2), pcs.catalog_version_no) AS suffix) AS dir, +lateral pg_ls_dir_recurse(dir) AS a; +</programlisting> + </para> + <indexterm> <primary>pg_ls_logdir</primary> </indexterm> diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 4511f9d658..e37ad20013 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1437,6 +1437,7 @@ REVOKE EXECUTE ON FUNCTION pg_stat_file(text,boolean) FROM public; REVOKE EXECUTE ON FUNCTION pg_ls_dir(text) FROM public; REVOKE EXECUTE ON FUNCTION pg_ls_dir(text,boolean,boolean) FROM public; REVOKE EXECUTE ON FUNCTION pg_ls_dir_metadata(text,boolean,boolean) FROM public; +REVOKE EXECUTE ON FUNCTION pg_ls_dir_recurse(text) FROM public; -- -- We also set up some things as accessible to standard roles. diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 408bd40256..6c3b6f1f5b 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -10813,6 +10813,12 @@ proallargtypes => '{text,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}', proargnames => '{dirname,name,size,modification,isdir}', prosrc => 'pg_ls_dir_metadata_1arg' }, +{ oid => '5034', descr => 'list all files in a directory recursively', + proname => 'pg_ls_dir_recurse', prorows => '10000', proretset => 't', + provolatile => 'v', prorettype => 'record', proargtypes => 'text', + proallargtypes => '{text,text,int8,timestamptz,bool}', + proargnames => '{dirname,name,size,modification,isdir}', proargmodes => '{i,o,o,o,o}', + prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.modification, a.isdir from ls, lateral pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" }, # hash partitioning constraint function { oid => '5028', descr => 'hash partition CHECK constraint', diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out index 64b1417fb8..4188d684f0 100644 --- a/src/test/regress/expected/misc_functions.out +++ b/src/test/regress/expected/misc_functions.out @@ -243,6 +243,20 @@ select * from pg_ls_dir_metadata('.') limit 0; ------+------+--------------+------- (0 rows) +-- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix +SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal'; + name | isdir +-----------------------+------- + pg_wal | t + pg_wal/archive_status | t +(2 rows) + +-- Check that expected columns are present +SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0; + name | size | modification | isdir +------+------+--------------+------- +(0 rows) + -- -- Test adding a support function to a subject function -- diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql index 372345720d..6041c4f3dc 100644 --- a/src/test/regress/sql/misc_functions.sql +++ b/src/test/regress/sql/misc_functions.sql @@ -76,6 +76,12 @@ select name, isdir from pg_ls_dir_metadata('.', false, false) where name='.'; -- -- Check that expected columns are present select * from pg_ls_dir_metadata('.') limit 0; +-- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix +SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal'; + +-- Check that expected columns are present +SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0; + -- -- Test adding a support function to a subject function -- -- 2.17.0
>From bf2e00ec7c3e1fda45d0d5d8cb34afd617cdc424 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Fri, 6 Mar 2020 17:23:51 -0600 Subject: [PATCH v15 08/10] pg_ls_logdir to ignore error if initial/top dir is missing --- src/backend/utils/adt/genfile.c | 2 +- src/test/regress/input/tablespace.source | 4 ++++ src/test/regress/output/tablespace.source | 7 +++++++ 3 files changed, 12 insertions(+), 1 deletion(-) diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c index afef2bf4ab..e8dc1d1be5 100644 --- a/src/backend/utils/adt/genfile.c +++ b/src/backend/utils/adt/genfile.c @@ -645,7 +645,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags) Datum pg_ls_logdir(PG_FUNCTION_ARGS) { - return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_COMMON); + return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_COMMON | LS_DIR_MISSING_OK); } /* Function to return the list of files in the WAL directory */ diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/input/tablespace.source index 0b9cfe615e..2a1268e17c 100644 --- a/src/test/regress/input/tablespace.source +++ b/src/test/regress/input/tablespace.source @@ -16,6 +16,10 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@'; -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir() SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist'; +-- This tests the missing_ok parameter. If that's not functioning, this would ERROR if the logdir doesn't exist yet. +-- The name='' condition is never true, so the function runs to completion but returns zero rows. +SELECT * FROM pg_ls_logdir() WHERE name='Does not exist'; + -- try setting and resetting some properties for the new tablespace ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1); ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true); -- fail diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source index 1c88e914e3..ba9a3fe29a 100644 --- a/src/test/regress/output/tablespace.source +++ b/src/test/regress/output/tablespace.source @@ -21,6 +21,13 @@ SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspa ------+------+--------------+------- (0 rows) +-- This tests the missing_ok parameter. If that's not functioning, this would ERROR if the logdir doesn't exist yet. +-- The name='' condition is never true, so the function runs to completion but returns zero rows. +SELECT * FROM pg_ls_logdir() WHERE name='Does not exist'; + name | size | modification | isdir +------+------+--------------+------- +(0 rows) + -- try setting and resetting some properties for the new tablespace ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1); ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true); -- fail -- 2.17.0
>From 0192275d6918d7a13789fd77a56b55927839881e Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Mon, 9 Mar 2020 21:56:21 -0500 Subject: [PATCH v15 09/10] pg_ls_*dir to return all the metadata from pg_stat_file.. ..but it doesn't seem worth factoring out the common bits, since stat_file doesn't return a name, so all the field numbers are off by one. NOTE, the atime is now shown where the mtime used to be. Need catversion bump --- doc/src/sgml/func.sgml | 30 ++++++++--------- src/backend/utils/adt/genfile.c | 29 ++++++++++------- src/include/catalog/pg_proc.dat | 34 ++++++++++---------- src/test/regress/expected/misc_functions.out | 16 ++++----- src/test/regress/output/tablespace.source | 8 ++--- 5 files changed, 59 insertions(+), 58 deletions(-) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 7c8b102610..88397ccddb 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21397,8 +21397,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); </entry> <entry><type>setof record</type></entry> <entry> - For each file in the log directory, list the file's name, size, last - modification time, and a boolean indicating if it is a directory. + For each file in the log directory, list the file and its metadata. Access is granted to members of the <literal>pg_monitor</literal> role and may be granted to other non-superuser roles. </entry> @@ -21409,8 +21408,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); </entry> <entry><type>setof record</type></entry> <entry> - For each file in the WAL directory, list the file's name, size, last - modification time, and a boolean indicating if it is a directory. + For each file in the WAL directory, list the file and its metadata. Access is granted to members of the <literal>pg_monitor</literal> role and may be granted to other non-superuser roles. </entry> @@ -21421,9 +21419,8 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); </entry> <entry><type>setof record</type></entry> <entry> - For each file in the WAL archive status directory, list the file's - name, size, last modification time, and a boolean indicating if it is a - directory. Access is granted to members of the + For each file in the WAL archive status directory, list the file and its metadata. + Access is granted to members of the <literal>pg_monitor</literal> role and may be granted to other non-superuser roles. </entry> @@ -21435,8 +21432,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); <entry><type>setof record</type></entry> <entry> For the temporary directory within <parameter>tablespace</parameter>, - list each file's name, size, last modification time, and a boolean - indicating if it is a directory. If <parameter>tablespace</parameter> + list each file and its metadata. If <parameter>tablespace</parameter> is not provided, the <literal>pg_default</literal> tablespace is used. Access is granted to members of the <literal>pg_monitor</literal> role and may be granted to other non-superuser roles. @@ -21522,8 +21518,8 @@ lateral pg_ls_dir_recurse(dir) AS a; </indexterm> <para> <function>pg_ls_logdir</function> lists each file in the log directory, - along with file's size, last modification time, and a boolean - indicating if the file is a directory. By default, only superusers + along with the metadata columns returned by <function>pg_stat_file</function>. + By default, only superusers and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. Filenames beginning with a dot and special file types are not shown. @@ -21534,8 +21530,8 @@ lateral pg_ls_dir_recurse(dir) AS a; </indexterm> <para> <function>pg_ls_waldir</function> lists each file in the WAL directory, - along with the file's size, last modification time, and a boolean - indicating if the file is a directory. By default, only superusers + along with the metadata columns returned by <function>pg_stat_file</function>. + By default, only superusers and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. @@ -21547,8 +21543,8 @@ lateral pg_ls_dir_recurse(dir) AS a; </indexterm> <para> <function>pg_ls_archive_statusdir</function> lists each file in the WAL - archive status directory, along with the file's size, last modification - time, and a boolean indicating if the file is a directory. By default, only + archive status directory, along with the metadata columns returned by + <function>pg_stat_file</function>. By default, only superusers and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. @@ -21561,8 +21557,8 @@ lateral pg_ls_dir_recurse(dir) AS a; <para> <function>pg_ls_tmpdir</function> lists each file in the temporary file directory for the specified <parameter>tablespace</parameter>, along with - its size, last modified time (mtime) and a boolean indicating if the file is a - directory. Directories are used for temporary files shared by parallel + the metadata columns returned by <function>pg_stat_file</function>. + Directories are used for temporary files shared by parallel processes. If <parameter>tablespace</parameter> is not provided, the <literal>pg_default</literal> tablespace is used. By default only superusers and members of the <literal>pg_monitor</literal> diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c index e8dc1d1be5..4d90f37a21 100644 --- a/src/backend/utils/adt/genfile.c +++ b/src/backend/utils/adt/genfile.c @@ -576,8 +576,8 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags) while ((de = ReadDir(dirdesc, dir)) != NULL) { - Datum values[4]; - bool nulls[4]; + Datum values[7]; + bool nulls[7]; char path[MAXPGPATH * 2]; struct stat attrib; @@ -616,24 +616,29 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags) continue; } + memset(nulls, 0, sizeof(nulls)); values[0] = CStringGetTextDatum(de->d_name); if (flags & LS_DIR_METADATA) { values[1] = Int64GetDatum((int64) attrib.st_size); - values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime)); - if (flags & LS_DIR_ISDIR) - { - values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode)); + values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_atime)); + values[3] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime)); + /* Unix has file status change time, while Win32 has creation time */ +#if !defined(WIN32) && !defined(__CYGWIN__) + values[4] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_ctime)); + nulls[5] = true; +#else + nulls[4] = true; + values[5] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_ctime)); +#endif + values[6] = BoolGetDatum(S_ISDIR(attrib.st_mode)); #ifdef WIN32 - /* Links should have isdir=false */ - if (pgwin32_is_junction(path)) - values[3] = BoolGetDatum(false); + /* Links should have isdir=false */ + if (pgwin32_is_junction(path)) + values[6] = BoolGetDatum(false); #endif - } } - memset(nulls, 0, sizeof(nulls)); - tuplestore_putvalues(tupstore, tupdesc, values, nulls); } diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 6c3b6f1f5b..d97ff8097f 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -10777,48 +10777,48 @@ { oid => '3353', descr => 'list files in the log directory', proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}', - proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_logdir' }, + proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_logdir' }, { oid => '3354', descr => 'list of files in the WAL directory', proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}', - proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_waldir' }, + proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_waldir' }, { oid => '5031', descr => 'list of files in the archive_status directory', proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', - proargtypes => '', proallargtypes => '{text,int8,timestamptz,bool}', - proargmodes => '{o,o,o,o}', proargnames => '{name,size,modification,isdir}', + proargtypes => '', proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', + proargmodes => '{o,o,o,o,o,o,o}', proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_archive_statusdir' }, { oid => '5029', descr => 'list files in the pgsql_tmp directory', proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}', - proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_noargs' }, + proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_tmpdir_noargs' }, { oid => '5030', descr => 'list files in the pgsql_tmp directory', proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => 'oid', - proallargtypes => '{oid,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}', - proargnames => '{tablespace,name,size,modification,isdir}', + proallargtypes => '{oid,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,o,o,o,o,o,o,o}', + proargnames => '{tablespace,name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_tmpdir_1arg' }, { oid => '5032', descr => 'list directory with metadata', proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool', - proallargtypes => '{text,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o}', - proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,modification,isdir}', + proallargtypes => '{text,bool,bool,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o,o,o,o}', + proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_dir_metadata' }, { oid => '5033', descr => 'list directory with metadata', proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => 'text', - proallargtypes => '{text,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}', - proargnames => '{dirname,name,size,modification,isdir}', + proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,o,o,o,o,o,o,o}', + proargnames => '{dirname,name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_dir_metadata_1arg' }, { oid => '5034', descr => 'list all files in a directory recursively', proname => 'pg_ls_dir_recurse', prorows => '10000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => 'text', - proallargtypes => '{text,text,int8,timestamptz,bool}', - proargnames => '{dirname,name,size,modification,isdir}', proargmodes => '{i,o,o,o,o}', - prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.modification, a.isdir from ls, lateral pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" }, + proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', + proargnames => '{dirname,name,size,access,modification,change,creation,isdir}', proargmodes => '{i,o,o,o,o,o,o,o}', + prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.access, a.modification, a.change, a.creation, a.isdir from ls, lateral pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" }, # hash partitioning constraint function { oid => '5028', descr => 'hash partition CHECK constraint', diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out index 4188d684f0..cce84a60a9 100644 --- a/src/test/regress/expected/misc_functions.out +++ b/src/test/regress/expected/misc_functions.out @@ -157,8 +157,8 @@ select count(*) > 0 as ok from (select pg_ls_waldir()) ss; -- Test not-run-to-completion cases. select * from pg_ls_waldir() limit 0; - name | size | modification | isdir -------+------+--------------+------- + name | size | access | modification | change | creation | isdir +------+------+--------+--------------+--------+----------+------- (0 rows) select count(*) > 0 as ok from (select * from pg_ls_waldir() limit 1) ss; @@ -222,8 +222,8 @@ ERROR: could not open directory "does not exist": No such file or directory -- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet -- The name='' condition is never true, so the function runs to completion but returns zero rows. select * from pg_ls_tmpdir() where name='Does not exist'; - name | size | modification | isdir -------+------+--------------+------- + name | size | access | modification | change | creation | isdir +------+------+--------+--------------+--------+----------+------- (0 rows) select name, isdir from pg_ls_dir_metadata('.') where name='.'; @@ -239,8 +239,8 @@ select name, isdir from pg_ls_dir_metadata('.', false, false) where name='.'; -- -- Check that expected columns are present select * from pg_ls_dir_metadata('.') limit 0; - name | size | modification | isdir -------+------+--------------+------- + name | size | access | modification | change | creation | isdir +------+------+--------+--------------+--------+----------+------- (0 rows) -- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix @@ -253,8 +253,8 @@ SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal'; -- Check that expected columns are present SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0; - name | size | modification | isdir -------+------+--------------+------- + name | size | access | modification | change | creation | isdir +------+------+--------+--------------+--------+----------+------- (0 rows) -- diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source index ba9a3fe29a..1e1e02b589 100644 --- a/src/test/regress/output/tablespace.source +++ b/src/test/regress/output/tablespace.source @@ -17,15 +17,15 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@'; -- The name='' condition is never true, so the function runs to completion but returns zero rows. -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir() SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist'; - name | size | modification | isdir -------+------+--------------+------- + name | size | access | modification | change | creation | isdir +------+------+--------+--------------+--------+----------+------- (0 rows) -- This tests the missing_ok parameter. If that's not functioning, this would ERROR if the logdir doesn't exist yet. -- The name='' condition is never true, so the function runs to completion but returns zero rows. SELECT * FROM pg_ls_logdir() WHERE name='Does not exist'; - name | size | modification | isdir -------+------+--------------+------- + name | size | access | modification | change | creation | isdir +------+------+--------+--------------+--------+----------+------- (0 rows) -- try setting and resetting some properties for the new tablespace -- 2.17.0
>From 9091a9138a8c2622e84ea2a06e922b8cb703be62 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Tue, 31 Mar 2020 14:40:34 -0500 Subject: [PATCH v15 10/10] pg_ls_* to show file type and show special files --- src/backend/utils/adt/genfile.c | 52 ++++++++++++-------- src/include/catalog/pg_proc.dat | 38 +++++++------- src/test/regress/expected/misc_functions.out | 40 +++++++-------- src/test/regress/output/tablespace.source | 8 +-- src/test/regress/sql/misc_functions.sql | 6 +-- 5 files changed, 77 insertions(+), 67 deletions(-) diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c index 4d90f37a21..1fdb2bf280 100644 --- a/src/backend/utils/adt/genfile.c +++ b/src/backend/utils/adt/genfile.c @@ -38,7 +38,7 @@ static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags); -#define LS_DIR_ISDIR (1<<0) /* Show column: isdir */ +#define LS_DIR_TYPE (1<<0) /* Show column: type */ #define LS_DIR_METADATA (1<<1) /* Show columns: mtime, size */ #define LS_DIR_MISSING_OK (1<<2) /* Ignore ENOENT if the toplevel dir is missing */ #define LS_DIR_SKIP_DOT_DIRS (1<<3) /* Do not show . or .. */ @@ -53,7 +53,7 @@ static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags #define LS_DIR_HISTORIC (LS_DIR_SKIP_DIRS|LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA) /* Shortcut for common behavior */ -#define LS_DIR_COMMON (LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA) +#define LS_DIR_COMMON (LS_DIR_SKIP_HIDDEN|LS_DIR_METADATA) /* * Convert a "text" filename argument to C string, and check it's allowable. @@ -367,6 +367,26 @@ pg_read_binary_file_all(PG_FUNCTION_ARGS) return pg_read_binary_file(fcinfo); } +/* Return a character indicating the type of file, or '?' if unknown type */ +static char +get_file_type(mode_t mode, const char *path) +{ + if (S_ISREG(mode)) + return '-'; + + if (S_ISDIR(mode)) + return 'd'; +#ifndef WIN32 + if (S_ISLNK(mode)) + return 'l'; +#else + if (pgwin32_is_junction(path)) + return 'l'; +#endif + + return '?'; +} + /* * stat a file */ @@ -414,7 +434,7 @@ pg_stat_file(PG_FUNCTION_ARGS) TupleDescInitEntry(tupdesc, (AttrNumber) 5, "creation", TIMESTAMPTZOID, -1, 0); TupleDescInitEntry(tupdesc, (AttrNumber) 6, - "isdir", BOOLOID, -1, 0); + "type", CHAROID, -1, 0); BlessTupleDesc(tupdesc); memset(isnull, false, sizeof(isnull)); @@ -430,12 +450,7 @@ pg_stat_file(PG_FUNCTION_ARGS) isnull[3] = true; values[4] = TimestampTzGetDatum(time_t_to_timestamptz(fst.st_ctime)); #endif - values[5] = BoolGetDatum(S_ISDIR(fst.st_mode)); -#ifdef WIN32 - /* Links should have isdir=false */ - if (pgwin32_is_junction(filename)) - values[5] = BoolGetDatum(false); -#endif + values[5] = CharGetDatum(get_file_type(fst.st_mode, filename)); tuple = heap_form_tuple(tupdesc, values, isnull); @@ -501,10 +516,10 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags) MemoryContext oldcontext; TypeFuncClass tuptype ; - /* isdir depends on metadata */ - Assert(!(flags&LS_DIR_ISDIR) || (flags&LS_DIR_METADATA)); - /* Unreasonable to show isdir and skip dirs */ - Assert(!(flags&LS_DIR_ISDIR) || !(flags&LS_DIR_SKIP_DIRS)); + /* type depends on metadata */ + Assert(!(flags&LS_DIR_TYPE) || (flags&LS_DIR_METADATA)); + /* Unreasonable to show type and skip dirs XXX */ + Assert(!(flags&LS_DIR_TYPE) || !(flags&LS_DIR_SKIP_DIRS)); /* check the optional arguments */ if (PG_NARGS() == 3) @@ -631,12 +646,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags) nulls[4] = true; values[5] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_ctime)); #endif - values[6] = BoolGetDatum(S_ISDIR(attrib.st_mode)); -#ifdef WIN32 - /* Links should have isdir=false */ - if (pgwin32_is_junction(path)) - values[6] = BoolGetDatum(false); -#endif + values[6] = CharGetDatum(get_file_type(attrib.st_mode, path)); } tuplestore_putvalues(tupstore, tupdesc, values, nulls); @@ -718,7 +728,7 @@ pg_ls_dir_metadata(PG_FUNCTION_ARGS) char *dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0)); return pg_ls_dir_files(fcinfo, dirname, - LS_DIR_METADATA | LS_DIR_SKIP_SPECIAL | LS_DIR_ISDIR); + LS_DIR_METADATA | LS_DIR_SKIP_SPECIAL | LS_DIR_TYPE); } /* @@ -733,5 +743,5 @@ pg_ls_dir_metadata_1arg(PG_FUNCTION_ARGS) char *dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0)); return pg_ls_dir_files(fcinfo, dirname, - LS_DIR_METADATA | LS_DIR_SKIP_SPECIAL | LS_DIR_ISDIR); + LS_DIR_METADATA | LS_DIR_SKIP_SPECIAL | LS_DIR_TYPE); } diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index d97ff8097f..8f19220456 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6107,16 +6107,16 @@ { oid => '2623', descr => 'get information about file', proname => 'pg_stat_file', provolatile => 'v', prorettype => 'record', proargtypes => 'text', - proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', + proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}', proargmodes => '{i,o,o,o,o,o,o}', - proargnames => '{filename,size,access,modification,change,creation,isdir}', + proargnames => '{filename,size,access,modification,change,creation,type}', prosrc => 'pg_stat_file_1arg' }, { oid => '3307', descr => 'get information about file', proname => 'pg_stat_file', provolatile => 'v', prorettype => 'record', proargtypes => 'text bool', - proallargtypes => '{text,bool,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', + proallargtypes => '{text,bool,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}', proargmodes => '{i,i,o,o,o,o,o,o}', - proargnames => '{filename,missing_ok,size,access,modification,change,creation,isdir}', + proargnames => '{filename,missing_ok,size,access,modification,change,creation,type}', prosrc => 'pg_stat_file' }, { oid => '2624', descr => 'read text from a file', proname => 'pg_read_file', provolatile => 'v', prorettype => 'text', @@ -10777,13 +10777,13 @@ { oid => '3353', descr => 'list files in the log directory', proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}', - proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_logdir' }, + proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}', proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{name,size,access,modification,change,creation,type}', prosrc => 'pg_ls_logdir' }, { oid => '3354', descr => 'list of files in the WAL directory', proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}', - proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_waldir' }, + proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}', proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{name,size,access,modification,change,creation,type}', prosrc => 'pg_ls_waldir' }, { oid => '5031', descr => 'list of files in the archive_status directory', proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', @@ -10793,32 +10793,32 @@ { oid => '5029', descr => 'list files in the pgsql_tmp directory', proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}', - proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_tmpdir_noargs' }, + proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}', proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{name,size,access,modification,change,creation,type}', prosrc => 'pg_ls_tmpdir_noargs' }, { oid => '5030', descr => 'list files in the pgsql_tmp directory', proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => 'oid', - proallargtypes => '{oid,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,o,o,o,o,o,o,o}', - proargnames => '{tablespace,name,size,access,modification,change,creation,isdir}', + proallargtypes => '{oid,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}', proargmodes => '{i,o,o,o,o,o,o,o}', + proargnames => '{tablespace,name,size,access,modification,change,creation,type}', prosrc => 'pg_ls_tmpdir_1arg' }, { oid => '5032', descr => 'list directory with metadata', proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool', - proallargtypes => '{text,bool,bool,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o,o,o,o}', - proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,access,modification,change,creation,isdir}', + proallargtypes => '{text,bool,bool,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}', proargmodes => '{i,i,i,o,o,o,o,o,o,o}', + proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,access,modification,change,creation,type}', prosrc => 'pg_ls_dir_metadata' }, { oid => '5033', descr => 'list directory with metadata', proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => 'text', - proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,o,o,o,o,o,o,o}', - proargnames => '{dirname,name,size,access,modification,change,creation,isdir}', + proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}', proargmodes => '{i,o,o,o,o,o,o,o}', + proargnames => '{dirname,name,size,access,modification,change,creation,type}', prosrc => 'pg_ls_dir_metadata_1arg' }, { oid => '5034', descr => 'list all files in a directory recursively', proname => 'pg_ls_dir_recurse', prorows => '10000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => 'text', - proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', - proargnames => '{dirname,name,size,access,modification,change,creation,isdir}', proargmodes => '{i,o,o,o,o,o,o,o}', - prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.access, a.modification, a.change, a.creation, a.isdir from ls, lateral pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" }, + proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}', + proargnames => '{dirname,name,size,access,modification,change,creation,type}', proargmodes => '{i,o,o,o,o,o,o,o}', + prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.access, a.modification, a.change, a.creation, a.type from ls, lateral pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.type='d') select * from ls" }, # hash partitioning constraint function { oid => '5028', descr => 'hash partition CHECK constraint', diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out index cce84a60a9..1bdcca16fc 100644 --- a/src/test/regress/expected/misc_functions.out +++ b/src/test/regress/expected/misc_functions.out @@ -157,8 +157,8 @@ select count(*) > 0 as ok from (select pg_ls_waldir()) ss; -- Test not-run-to-completion cases. select * from pg_ls_waldir() limit 0; - name | size | access | modification | change | creation | isdir -------+------+--------+--------------+--------+----------+------- + name | size | access | modification | change | creation | type +------+------+--------+--------------+--------+----------+------ (0 rows) select count(*) > 0 as ok from (select * from pg_ls_waldir() limit 1) ss; @@ -222,39 +222,39 @@ ERROR: could not open directory "does not exist": No such file or directory -- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet -- The name='' condition is never true, so the function runs to completion but returns zero rows. select * from pg_ls_tmpdir() where name='Does not exist'; - name | size | access | modification | change | creation | isdir -------+------+--------+--------------+--------+----------+------- + name | size | access | modification | change | creation | type +------+------+--------+--------------+--------+----------+------ (0 rows) -select name, isdir from pg_ls_dir_metadata('.') where name='.'; - name | isdir -------+------- - . | t +select name, type from pg_ls_dir_metadata('.') where name='.'; + name | type +------+------ + . | d (1 row) -select name, isdir from pg_ls_dir_metadata('.', false, false) where name='.'; -- include_dot_dirs=false - name | isdir -------+------- +select name, type from pg_ls_dir_metadata('.', false, false) where name='.'; -- include_dot_dirs=false + name | type +------+------ (0 rows) -- Check that expected columns are present select * from pg_ls_dir_metadata('.') limit 0; - name | size | access | modification | change | creation | isdir -------+------+--------+--------------+--------+----------+------- + name | size | access | modification | change | creation | type +------+------+--------+--------------+--------+----------+------ (0 rows) -- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix -SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal'; - name | isdir ------------------------+------- - pg_wal | t - pg_wal/archive_status | t +SELECT name, type FROM pg_ls_dir_recurse('.') WHERE type='d' AND name~'^pg_wal'; + name | type +-----------------------+------ + pg_wal | d + pg_wal/archive_status | d (2 rows) -- Check that expected columns are present SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0; - name | size | access | modification | change | creation | isdir -------+------+--------+--------------+--------+----------+------- + name | size | access | modification | change | creation | type +------+------+--------+--------------+--------+----------+------ (0 rows) -- diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source index 1e1e02b589..025c9709a1 100644 --- a/src/test/regress/output/tablespace.source +++ b/src/test/regress/output/tablespace.source @@ -17,15 +17,15 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@'; -- The name='' condition is never true, so the function runs to completion but returns zero rows. -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir() SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist'; - name | size | access | modification | change | creation | isdir -------+------+--------+--------------+--------+----------+------- + name | size | access | modification | change | creation | type +------+------+--------+--------------+--------+----------+------ (0 rows) -- This tests the missing_ok parameter. If that's not functioning, this would ERROR if the logdir doesn't exist yet. -- The name='' condition is never true, so the function runs to completion but returns zero rows. SELECT * FROM pg_ls_logdir() WHERE name='Does not exist'; - name | size | access | modification | change | creation | isdir -------+------+--------+--------------+--------+----------+------- + name | size | access | modification | change | creation | type +------+------+--------+--------------+--------+----------+------ (0 rows) -- try setting and resetting some properties for the new tablespace diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql index 6041c4f3dc..3874625ca1 100644 --- a/src/test/regress/sql/misc_functions.sql +++ b/src/test/regress/sql/misc_functions.sql @@ -69,15 +69,15 @@ select pg_ls_dir('does not exist'); -- fails with missingok=false -- The name='' condition is never true, so the function runs to completion but returns zero rows. select * from pg_ls_tmpdir() where name='Does not exist'; -select name, isdir from pg_ls_dir_metadata('.') where name='.'; +select name, type from pg_ls_dir_metadata('.') where name='.'; -select name, isdir from pg_ls_dir_metadata('.', false, false) where name='.'; -- include_dot_dirs=false +select name, type from pg_ls_dir_metadata('.', false, false) where name='.'; -- include_dot_dirs=false -- Check that expected columns are present select * from pg_ls_dir_metadata('.') limit 0; -- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix -SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal'; +SELECT name, type FROM pg_ls_dir_recurse('.') WHERE type='d' AND name~'^pg_wal'; -- Check that expected columns are present SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0; -- 2.17.0