How to only auto-restart BGW only on crash or _PG_init

Good afternoon! I would be grateful for some direction on how to use Background workers to have a worker automatically restart *only* in certain cases, i.e. on postmaster start (_PG_init) or a soft crash. I run into all sorts of trouble if I set bgw_restart_time to actually restart on sigterm,

Re: PHJ file leak.

On Fri, Mar 6, 2020 at 9:43 AM Tom Lane wrote: > Jeremy Finzel writes: > > We are hitting this leak in production on an 11.6 system for a query that > > is using a parallel hash join. Was this fix pushed in 11.7? I can't tell > > clearly from the release notes fo

Re: PHJ file leak.

On Fri, Nov 15, 2019 at 3:32 PM Thomas Munro wrote: > On Wed, Nov 13, 2019 at 9:52 PM Amit Kapila > wrote: > > On Wed, Nov 13, 2019 at 6:13 AM Kyotaro Horiguchi < > horikyota@gmail.com> wrote: > > > The phatch's shape looks better. Thanks. > > > > +1. LGTM as well. > > Thanks. Pushed. > >

Re: Index corruption / planner issue with one table in my pg 11.6 instance

On Tue, Dec 10, 2019 at 8:25 AM Jeremy Finzel wrote: > Is there a way for me to test this theory? I tried the following with no > change in behavior: > >1. Disable write load to table >2. Vacuum analyze table (not vac full) >3. Create index >4. Explain >

Re: Index corruption / planner issue with one table in my pg 11.6 instance

On Tue, Dec 10, 2019 at 12:09 AM Tom Lane wrote: > Yeah. The reported behavior can mostly be explained if we assume > that there's some HOT chain in the table that involves an update > of this particular column, so that if we build an index on that > column we see a broken HOT chain, but

Index corruption / planner issue with one table in my pg 11.6 instance

I have a table with about 7 million records. I had a query in which I needed 2 indexes added, one for a created timestamp field another for an id field; both very high cardinality. First I noticed the query would not use the timestamp index no matter what session config settings I used. I

Re: physical slot xmin dependency on logical slot?

> > I expect that you created the replica in a manner that preserved the > logical replication slot on it. You also had hot_standby_feedback enabled. > As per both you and Andres' replies, we wanted the backup to have the logical slots on it, because we wanted to allow decoding from the slots on

physical slot xmin dependency on logical slot?

We had a scenario today that was new to us. We had a logical replication slot that was severely far behind. Before dropping this logical slot, we made a physical point-in-time-recovery snapshot of the system with this logical slot. This logical slot was causing severe catalog bloat. We

Re: UPSERT on view does not find constraint by name

On Fri, Oct 18, 2019 at 3:42 AM Tom Lane wrote: > Jeremy Finzel writes: > > I'm not sure if this can be considered a bug or not, but it is perhaps > > unexpected. I found that when using a view that is simply select * from > > table, then doing INSERT ... ON C

UPSERT on view does not find constraint by name

I'm not sure if this can be considered a bug or not, but it is perhaps unexpected. I found that when using a view that is simply select * from table, then doing INSERT ... ON CONFLICT ON CONSTRAINT constraint_name on that view, it does not find the constraint and errors out. But it does find the

Re: BRIN index which is much faster never chosen by planner

Thank you for the thorough and thoughtful reply! Please see below. On Mon, Oct 14, 2019 at 3:48 PM David Rowley wrote: > Another thing which you might want to look at is the correlation > column in the pg_stats view for the rec_insert_time column. Previous > to 7e534adcd, BRIN index were

Re: BRIN index which is much faster never chosen by planner

> > The other issue is that the estimation of pages fetched using bitmap > heap scan is rather crude - but that's simply hard, and I don't think we > can fundamentally improve this. > I wanted to follow up on this specific issue. Isn't this the heart of the matter and a fundamental problem? If

Re: BRIN index which is much faster never chosen by planner

On Thu, Oct 10, 2019 at 6:13 PM Tomas Vondra wrote: > > So this seems like a combination of multiple issues. Firstly, the bitmap > index scan on rec_insert_time_brin_1000 estimate seems somewhat poor. It > might be worth increasing stats target on that column, or something like > that. Not sure,

Re: BRIN index which is much faster never chosen by planner

Dear Michael, On Thu, Oct 10, 2019 at 5:20 PM Michael Lewis wrote: > Since the optimizer is choosing a seq scan over index scan when it seems > like it has good row estimates in both cases, to me that may mean costs of > scanning index are expected to be high. Is this workload on SSD? Has the >

Re: BRIN index which is much faster never chosen by planner

On Thu, Oct 10, 2019 at 7:22 PM David Rowley wrote: > The planner might be able to get a better estimate on the number of > matching rows if the now() - interval '10 days' expression was > replaced with 'now'::timestamptz - interval '10 days'. However, care > would need to be taken to ensure the

BRIN index which is much faster never chosen by planner

Good Afternoon, I posted about this on another thread here , but the topic was not precisely planner issues, so I wanted to post it here. I am running Postgres 11.5. I have a table that is

pg_regress restart cluster?

Hello! I am using the basic extension building infrastructure with sql and expected files, but what I want to test is changing a config setting and then restarting the cluster with shared_preload_libraries in place. Is there a canonical way to do this or does anyone have any examples of this? I

Configuring bgw_restart_time

I am working on an extension that uses background workers, and interested in adding code for it to auto-restart after a crash or server restart (similar to as-coded in worker_spi). But I'm also interested in being able to configure bgw_restart_time using a GUC without having to restart the

Re: proposal - patch: psql - sort_by_size

On Fri, Jun 28, 2019 at 10:13 AM Pavel Stehule wrote: > Hi > > I returned to possibility to sort output of \d* and \l by size. There was > more a experiments in this area, but without success. Last patch was > example of over engineering, and now, I try to implement this feature > simply how it

Docs for refresh materialized view concurrently

views notes that they can only be LOGGED. This should be noted, or at least it should be noted that one cannot create an UNLOGGED materialized view in the same place it says that one cannot create a temporary one (under Description in CREATE MATERIALIZED VIEW). Thanks! Jeremy Finzel

Question about new pg12 feature no-rewrite timestamp to timestamptz conversion

I have a question about this (really exciting) feature coming in pg12: Allow ALTER TABLE .. SET DATA TYPE timestamp/timestamptz to avoid a table rewrite when the session time zone is UTC (Noah Misch) In the UTC time zone, the data types are binary compatible. We actually want to migrate all of

Re: Automated way to find actual COMMIT LSN of subxact LSN

> > I find this to be unactionably vague. What does it mean to claim "an > LSN is visible"? An LSN might not even point to a WAL record, or it > might point to one that has nontransactional effects. Moreover, any > behavior of this sort would destroy what I regard as a bedrock property > of

Re: Automated way to find actual COMMIT LSN of subxact LSN

> > It would seem like what you're asking for is to continue until the commit > of the parent transaction, not just the next commit after the subcommit. > Otherwise (if that's an unrelated xact) the subxact would still not be > committed, so that you might as well have stopped short of it. >

Re: Automated way to find actual COMMIT LSN of subxact LSN

> > If recovery_target_inclusive were able to take the third value > "xact", is it exactly what you want? > > And is it acceptable? > Yes, that would be exactly what I would want. It would work to have a 3rd value for recovery_target_inclusive, although perhaps it's debatable that instead, it

Automated way to find actual COMMIT LSN of subxact LSN

I want to build automation to recover a database to a specific LSN *inclusive*, even if that LSN is from a subtransaction. The problem I am facing is that I know what specific LSN wrote a row on a remote system, but if I create a recovery.conf file with: recovery_target_lsn = '95F/BBA36DF8' and

relcache reference leak with pglogical replication to insert-only partitioned table?

I understand it's not fully supported to replicate to a differently partitioned setup on a subscriber with either pglogical or the native logical replication, however I also know that INSERT triggers can be fired in replication mode. I have an insert-only OLTP table that I want partitioned only

Implicit timezone conversion replicating from timestamp to timestamptz?

We are working to migrate several large tables from the timestamp to the timestamptz data type by using logical replication (so as to avoid long downtime for type conversions). We are using pglogical but curious if what I share below applies to native logical replication as well. Both source and

Re: No such file or directory in pg_replslot

On Sat, Dec 8, 2018 at 1:21 PM Jeremy Finzel wrote: > That doesn't indicate an error. You need to provide more details what >> made you consider things wedged... >> >> Andres >> > > Thank you very much for the reply. We typically see no visible > replicati

Re: No such file or directory in pg_replslot

> > That doesn't indicate an error. You need to provide more details what > made you consider things wedged... > > Andres > Thank you very much for the reply. We typically see no visible replication delay over 5 minutes ever. Today we saw a delay of over 3 hours, and no obvious increase in

No such file or directory in pg_replslot

I don't know if this applies only to pglogical or logical decoding in general. This is on a 9.6.10 provider running pglogical 2.2.0. Subscriber has same versions. We had a replication delay situation this morning, which I think may have been due to a really long transaction but I've yet to

How do I get altered object from GRANT event trigger?

I noticed that there is no information for GRANT in pg_event_trigger_ddl_commands(). I am trying to determine which table/schema is being altered. Is there any way to do this either with built-in functionality, or extending the current code in event_trigger.c? Is the reason for this limitation

worker_spi shouldn't execute again on sigterm

I noticed that the way the test module worker_spi is written, it will execute the main loop SQL one more time after it gets a sigterm, THEN exit 1. This was surprising to me where I used this module as a pattern for my own background worker as I would have thought it should bail immediately

Re: Race condition in WaitForBackgroundWorkerStartup

On Tue, Nov 13, 2018 at 6:17 AM Amit Kapila wrote: > On Mon, Nov 12, 2018 at 11:55 PM Jeremy Finzel wrote: > > > > I believe I found a race condition in WaitForBackgroundWorkerStartup in > the case where it encounters an ERROR during startup. I found that > depending on th

Race condition in WaitForBackgroundWorkerStartup

I believe I found a race condition in WaitForBackgroundWorkerStartup in the case where it encounters an ERROR during startup. I found that depending on the speed of the system, it will unreliably return either status BGWH_STOPPED or BGWH_STARTED. But I can reliably reproduce getting BGWH_STOPPED

Re: Changes to error handling for background worker initialization?

On Mon, Oct 22, 2018 at 9:36 AM Jeremy Finzel wrote: > Hello - > > I have an extension that uses background workers. I pass a database oid > as an argument in order to launch the worker using function > BackgroundWorkerInitializeConnectionByOid. In one of my regression tests >

Changes to error handling for background worker initialization?

Hello - I have an extension that uses background workers. I pass a database oid as an argument in order to launch the worker using function BackgroundWorkerInitializeConnectionByOid. In one of my regression tests that was written, I intentionally launch the worker with an invalid oid. In

Re: Proposal for disk quota feature

On Fri, Sep 21, 2018 at 9:21 AM Hubert Zhang wrote: > just fast reaction - why QUOTA object? >> Isn't ALTER SET enough? >> Some like >> ALTER TABLE a1 SET quote = 1MB; >> ALTER USER ... >> ALTER SCHEMA .. >> New DDL commans looks like too hard hammer . > > > It's an option. Prefer to consider

Re: Delta Materialized View Refreshes?

> > > Could you apply something similar using triggers? > One question would be how PG would identify changes to existing rows - > using the replication facilities to essentially replicate into the view? > This would be quite tricky I reckon. Otherwise a change to the underlying > table may not

Logical replication to partitioned subscriber

Are there any plans to support logical replication to a table which is only partitioned on the subscriber? We are very interested in such a feature for a couple use cases. We are using logical replication for archiving data and we only want it partitioned on the archive server, and we want it to

Delta Materialized View Refreshes?

To me the biggest disadvantage of using a materialized view instead of a table delta load for an ETL process, for example, is that the materialized view is very inefficient in terms of write overhead, for of course it rewrites the whole set of data in the table, meaning heap, indexes, WAL and

Re: Add --include-table-data-where option to pg_dump, to export only a subset of table data

> > Just skimming the thread but I'd have to say being able to leverage > pg_dump's dependency resolution is a major reason for adding features to it > instead sticking to writing psql scripts. This feature in a multi-tenant > situation is something with, I suspect, reasonably wide appeal. >

Re: Add --include-table-data-where option to pg_dump, to export only a subset of table data

On Mon, May 21, 2018 at 6:34 AM Carter Thaxton wrote: > Many times I've wanted to export a subset of a database, using some sort > of row filter condition on some of the large tables. E.g. copying a > production database to a staging environment, but with some time series > data only from the

Re: Some pgq table rewrite incompatibility with logical decoding?

> > Interesting. > > So you were running 9.6.9 before, it triggered the issue (and was not > able to recover). You took a filesystem snapshot, started a 9.6.10 on > the snapshot, and it recovered without hitting the issue? > I am resposting this to the list and not only to Tomas. Tomas, I can’t

Re: Some pgq table rewrite incompatibility with logical decoding?

> > We have hit this error again, and we plan to snapshot the database as to > be able to do whatever troubleshooting we can. > I am happy to report that we were able to get replication working again by running snapshots of the systems in question on servers running the latest point release

Re: Some pgq table rewrite incompatibility with logical decoding?

> > Jeremy, are you able to reproduce the issue locally, using pgq? That would >> be very valuable. > > Tomas et al: We have hit this error again, and we plan to snapshot the database as to be able to do whatever troubleshooting we can. If someone could provide me guidance as to what exactly you

Re: Repeatable Read Isolation in SQL running via background worker

On Tue, Aug 14, 2018 at 11:18 AM, Robert Haas wrote: > On Mon, Aug 13, 2018 at 10:52 AM, Jeremy Finzel wrote: > > On Thu, Aug 9, 2018 at 4:34 PM, Jeremy Finzel wrote: > >> I am using worker_spi as a model to run a SQL statement inside a > >> background worker. From

Re: Some pgq table rewrite incompatibility with logical decoding?

> > I don't think that's true, for two reasons. > > Firstly, I don't think pgq updates catalogs directly, it simply truncates > the queue tables when rotating them (which updates the relfilenode in > pg_class, of course). > > Secondly, we're occasionally seeing this on systems that do not use pgq,

Re: Repeatable Read Isolation in SQL running via background worker

On Thu, Aug 9, 2018 at 4:34 PM, Jeremy Finzel wrote: > I am using worker_spi as a model to run a SQL statement inside a > background worker. From my browsing of the Postgres library, I believe > that if I want repeatable read isolation level, the proper way for me to > attain th

Repeatable Read Isolation in SQL running via background worker

I am using worker_spi as a model to run a SQL statement inside a background worker. From my browsing of the Postgres library, I believe that if I want repeatable read isolation level, the proper way for me to attain this is to add this line after StartTransactionCommand() in worker_spi_main:

Re: Auditing via logical decoding

On Fri, Jul 27, 2018 at 5:41 AM Philip Scott wrote: > Hi Postgres Hackers, > > We have been using our own trigger-based audit system at my firm > successfully for some years, but the performance penalty is starting to > grate a bit and so I have been tasked with seeing if we can make use of >

Using test_ddl_deparse as an extension

I am interested in using the exact functionality in test_ddl_deparse which provides the sub command types for alter table statements. I would prefer to leverage this code which has already been vetted by the community, and also not just duplicate this code myself, or package it up myself. It

Re: Early WIP/PoC for inlining CTEs

On Tue, Jul 24, 2018 at 5:28 PM Andrew Gierth wrote: > About a year ago I was briefly in discussion/collaboration with Adam Sah > regarding the topic of inlining CTEs into the query rather than treating > them as optimization barriers. We didn't take it very far (he sent me > some stuff, I wrote

Re: Background worker/idle sessions and caching

On Wed, Jul 18, 2018 at 3:05 PM, Tom Lane wrote: > Jeremy Finzel writes: > > I have a background worker running SQL functions, and I believe I have > > noticed that when I do things like change function definitions, or even > add > > tables, the background worker doe

Background worker/idle sessions and caching

I have a background worker running SQL functions, and I believe I have noticed that when I do things like change function definitions, or even add tables, the background worker does not pick up the schema changes until I restart the worker. Is this expected behavior? If I use background workers

Re: Logical decoding from promoted standby with same replication slot

On Fri, Jul 13, 2018 at 2:30 PM, Jeremy Finzel wrote: > Hello - > > We are working on several DR scenarios with logical decoding. Although we > are using pglogical the question we have I think is generally applicable to > logical replication. > > Say we have need to drop

Logical decoding from promoted standby with same replication slot

Hello - We are working on several DR scenarios with logical decoding. Although we are using pglogical the question we have I think is generally applicable to logical replication. Say we have need to drop a logical replication slot for some emergency reason on the master, but we don't want to

Some pgq table rewrite incompatibility with logical decoding?

I am hoping someone here can shed some light on this issue - I apologize if this isn't the right place to ask this but I'm almost some of you all were involving in pgq's dev and might be able to answer this. We are actually running 2 replication technologies on a few of our dbs, skytools and

Re: found xmin from before relfrozenxid on pg_catalog.pg_authid

On Tue, Jun 19, 2018 at 8:26 AM Matheus de Oliveira < matioli.math...@gmail.com> wrote: > Hello friends. > > On Tue, Jun 12, 2018 at 3:31 PM, Andres Freund wrote: > >> >> On 2018-06-11 17:39:14 -0700, Andres Freund wrote: >> > I plan to go over the change again tomorrow, and then push. Unless >>

Re: found xmin from before relfrozenxid on pg_catalog.pg_authid

On Fri, May 25, 2018 at 3:37 PM, Andres Freund wrote: > Hi, > > Moving discussion to -hackers. Tom, I think you worked most with this > code, your input would be appreciated. > > Original discussion is around: > http://archives.postgresql.org/message-id/20180524211311. >

Including SQL files in extension scripts

In writing extension update scripts, I find it to be really difficult to grok diffs for example in changed view or function definitions when a new extension script has to include the whole definition in a new file. I want to rather use separate files for these objects, then use something like

Re: Passing current_database to BackgroundWorkerInitializeConnection

On Mon, Apr 2, 2018 at 2:27 PM, Andres Freund <and...@anarazel.de> wrote: > Hi, > > On 2018-04-02 14:24:53 -0500, Jeremy Finzel wrote: > > Thank you, this makes sense. However, how can this be done since I can > > only pass one argument to bgw_main? Is there

Re: Passing current_database to BackgroundWorkerInitializeConnection

On Fri, Mar 30, 2018 at 5:37 PM, Andres Freund <and...@anarazel.de> wrote: > > > On March 30, 2018 3:16:31 PM PDT, Jeremy Finzel <finz...@gmail.com> wrote: > >> What do you mean with "current database"? Before you > >> BackgroundWorkerInitializeC

Re: Passing current_database to BackgroundWorkerInitializeConnection

> What do you mean with "current database"? Before you > BackgroundWorkerInitializeConnection() there is no such thing? My module is based directly off the worker_spi example. The worker is dynamically launched via SQL command. But in the worker_spi example, the database postgres is just

Passing current_database to BackgroundWorkerInitializeConnection

I am having trouble figuring out the right way to do this, clearly missing something obvious. I am simply trying to pass the current database to BackgroundWorkerInitializeConnection, but MyDatabaseId is showing as 0, and I am getting this error in the running of function get_database_name here at

Re: Feature Request - DDL deployment with logical replication

On Fri, Mar 30, 2018 at 2:26 AM, Konstantin Knizhnik < k.knizh...@postgrespro.ru> wrote: > > Our team in PostgresPro is also very interested in this discussion, > because we are using logical decoding in multimaster. > Right now in multimaster DDLs are replicated in this same way as in >

Re: Feature Request - DDL deployment with logical replication

On Fri, Mar 30, 2018 at 10:16 AM, Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > On 3/29/18 13:21, Jeremy Finzel wrote: > > Although we are thrilled with some of the features already in logical > > replication, this missing feature is the #1 reason that we d

Feature Request - DDL deployment with logical replication

Hello! I have not seen much discussion about what the plans are for being able to manage schema changes when using logical replication. In our own infrastructure, mechanisms that have been provided to manage DDL statements at the same transactional point as they happen on the master have been

Re: worker_spi.naptime in worker_spi example

On Thu, Mar 15, 2018 at 3:54 PM, Chapman Flack <c...@anastigmatix.net> wrote: > On 03/15/2018 04:19 PM, Jeremy Finzel wrote: > > > Thank you. I did see worker_spi_naptime definition, but I wanted to > > pass worker_spi_naptime > > as an argument instead into worker_

Re: worker_spi.naptime in worker_spi example

On Thu, Mar 15, 2018 at 12:21 PM, Chapman Flack <c...@anastigmatix.net> wrote: > On 03/15/2018 10:59 AM, Jeremy Finzel wrote: > > > WaitLatch function the same variable worker_spi_naptime without > referencing > > the configuration created in _PG_init. Am I missing

worker_spi.naptime in worker_spi example

Hello! I am trying to create a background worker where I can pass in via SQL interface the naptime. I'm not sure how to do this. In the worker_spi example, _PG_init creates a configuration using the static variable worker_spi_naptime, but then it appears to call in worker_spi_main in the

Re: using worker_spi as pattern

On Fri, Mar 9, 2018 at 12:34 AM, Michael Paquier <mich...@paquier.xyz> wrote: > On Thu, Mar 08, 2018 at 11:04:20PM -0600, Jeremy Finzel wrote: > > Since you mention, can anyone elaborate further on the memory leak danger > > here? > > > > Line 193 in src/test/mod

using worker_spi as pattern

Hello - I have compiled and installed the extension worker_spi. I also launched the process via SELECT worker_spi_launch(1); I see this in pg_stat_activity: WITH deleted AS (DELETE FROM schema1.counted WHERE type = 'delta' RETURNING value), total AS (SELECT coalesce(sum(value), 0) as sum FROM

Identifying ALTER TABLE "sub-command"

Hello - I have found that in leveraging the parser code to decode DDL SQL, it is very easy to get which type of general command is being issued with CreateCommandTag(parsetree). However, is there a way (or a starting point) to identify the sub-command as well i.e. ENABLE TRIGGER, ADD FOREIGN

Re: Deadlock in multiple CIC.

> > Since the purpose of CIC is to build an index with minimal impact on other > users, I think wanting to use it in concurrent cases might be rather rare. > In a maintenance window, I wouldn't want to use CIC because it is slower > and I'd rather just hold the stronger lock and do it fast, and as

Re: Deadlock in multiple CIC.

On Tue, Dec 26, 2017 at 11:23 AM, Jeff Janes wrote: > On Tue, Dec 26, 2017 at 8:31 AM, Alvaro Herrera > wrote: > >> Jeff Janes wrote: >> > c3d09b3bd23f5f6 fixed it so concurrent CIC would not deadlock (or at >> least >> > not as reliably as before)

Re: Backfill bgworker Extension?

On Sat, Dec 16, 2017 at 8:31 AM Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > On 12/15/17 23:50, Jeremy Finzel wrote: > > The common ground is some column in some table needs to be bulk updated. > > I may not be explaining well, but in our environment we h

Re: Backfill bgworker Extension?

On Tue, Dec 12, 2017 at 2:26 PM Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > On 12/12/17 13:03, Jeremy Finzel wrote: > > To be clear, what I mean is batch updating a large set of data in small > > pieces so as to avoid things like lock contention

Backfill bgworker Extension?

One of our challenges we have is that our engineers have written frameworks to backfill data in several different DSLs, and every time they adopt a new language, they maybe need to write another one. To be clear, what I mean is batch updating a large set of data in small pieces so as to avoid

Re: proposal: alternative psql commands quit and exit

Someone mentioned about needing to read the documentation of vim to learn how to exit the program. I personally think exactly the same applies here, and am a bit surprised at the depth of discussion over this. When I first was new to cli programs, the only "standard" way to exit a program I

Re: Testing Extension Upgrade Paths

On Mon, Dec 11, 2017 at 7:55 PM, Craig Ringer <cr...@2ndquadrant.com> wrote: > On 12 December 2017 at 07:25, Michael Paquier <michael.paqu...@gmail.com> > wrote: > >> On Tue, Dec 12, 2017 at 2:55 AM, Jeremy Finzel <finz...@gmail.com> wrote: >> > I

Testing Extension Upgrade Paths

Hello - I understand how to setup a regression suite for a postgres extension, but what I'm not clear on from the docs is if there is a pattern that exists for testing not only the latest version of an extension, but also an upgraded previous version. For example, I am developing version 1.1,

How to use set/reset role in contrib_regression test?

LE; SELECT CURRENT_ROLE; current_user - test_pgl_ddl_deploy (1 row) SET SESSION AUTHORIZATION DEFAULT; RESET ROLE; SELECT CURRENT_ROLE; current_user - test_pgl_ddl_deploy (1 row) What am I missing here? Any comments much appreciated. Thanks, Jeremy Finzel