Re: [ADMIN] Help desperately needed: reoccurring invalid page
Mauri, I'm no expert but I can tell you what I know. We have experienced this same problem, and I asked a similar question not too long ago. - This is probably due to some sort of I/O problem. In our case, it occurs very frequently on reboots, and it appears to be related to dirty buffers not being flushed out when the partition is unmounted. Look into a newer kernel, and/or newer/different filesystem (we have seen this in various flavors of the 2.4.6 kernel with XFS, JFS and EXT3). - If you have invalid page headers, you should take a look at the run-time parameter zero_damaged_pages. You can use this to fix up a table (losing the data on the bad pages). - We have not found a way around the missing log file problem. This appears to end up locking some rows forever. Others on this list may be able to tell you about tools for examining/fixing corrupt tables and log files. --Ian ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [ADMIN] Autovacuum Question
I just happened to be reading this page from the 8.1 docs: "The autovacuum daemon, when enabled, runs every autovacuum_naptime seconds and determines which database to process. Any database which is close to transaction ID wraparound is immediately processed. In this case, autovacuum issues a database-wide VACUUM call, or VACUUM FREEZE if it's a template database, and then terminates. If no database fulfills this criterion, the one that was least recently processed by autovacuum is chosen. In this case each table in the selected database is checked, and individual VACUUM or ANALYZE commands are issued as needed." -Original Message-From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]On Behalf Of Chris HooverSent: Tuesday, May 16, 2006 1:27 PMTo: pgsql-admin@postgresql.orgSubject: [ADMIN] Autovacuum QuestionPostgreSQL 8.1.3Question on autovacuum.autovacuum_naptime (integer) Specifies the delay between activity rounds for the autovacuum subprocess. In each round the subprocess examines one database and issues VACUUM and ANALYZE commands as needed for tables in that database. The delay is measured in seconds, and the default is 60. This option can be set at server start or in the postgresql.conf file.Does this mean that each run of autovacuum will vacuum/analyze all tables that need work in the database that autovacuum is looking at? The reason I'm asking is that I have ~172 non system tables in each database, so if autovacuum only does one table per run, I don't think it will ever get caught up (since there are almost 35k tables in total). Also, with over 200 databases, does this mean that a database will only get checked once about every 3.5 hours, or does autovacuum run on the database that needs the vacuuming the most?Thanks,Chris
[ADMIN] VACUUM and read-mostly tables
I have a near-real-time system writing into a Postgres 7.4.2 database on the order of 340 million rows per day in about 300 million transactions. So we quickly bump up against the XID wrap-around issue. To address this, we divide the tables into 24-hour periods. Once we roll over to a new period, the old tables are read-mostly. We then run vacuum every 24 hours. The problem is that we are writing rows every 1/15 second, 24x7. There is no down time. I'm wondering if there is any way to avoid vacuuming the old tables over and over. The documentation seems to indicate that a full vacuum is needed to avoid XID wrap-around. Can vacuum freeze help me? Thanks, --Ian ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [ADMIN] VACUUM and read-mostly tables
On Tue, 2005-04-05 at 00:41, Jim C. Nasby wrote: > We'll only answer if you do a write-up on your database. :P > > Seriously, those are some seriously big numbers. What else is the > database doing? What hardware is it running on? We run on a dual 3.2GHz P4 with 2GB RAM, but are still finalizing the storage hardware. We've tried various flavors of RAID, filesystems and volume management (and are anxious to try out tablespaces in 8). We've found fragmentation to be our largest limiting factor. XFS helps with that, and seems to provide the highest sustained throughput on raw tables, but its not the end of the story since fragmentation is still high. --Ian ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [ADMIN] VACUUM and read-mostly tables
On Tue, 2005-04-05 at 02:42, Tom Lane wrote: > "Ian Westmacott" <[EMAIL PROTECTED]> writes: > > The problem is that we are writing rows every 1/15 second, 24x7. There > > is no down time. I'm wondering if there is any way to avoid vacuuming > > the old tables over and over. The documentation seems to indicate that > > a full vacuum is needed to avoid XID wrap-around. Can vacuum freeze > > help me? > > VACUUM FULL isn't really relevant. VACUUM FREEZE on a particular table > should "fix" that table permanently, as long as you don't make any more > changes to it. Keep in mind though that you still have to vacuum the > system catalogs often enough to avoid wraparound in them. The real risk > here is in overlooking any one table. You should probably use some kind > of automated vacuum driver ... have you looked at pg_autovacuum? We did look at pg_autovacuum. We found the cost of row- level statistics too high to be workable. Looking at the pg_autovacuum code, it appears that a full vacuum is performed whenever the XIDs get old enough. The problem is that it is unpredictable when this will occur. So we have started work on our own automated driver, primarily for the analyze side of things at the moment, but I would like to extend it to vacuum as well. For example, we could update the row-level statistics on a less frequent basis than every insert. But the question is whether vacuum freezing tables will help me reduce the frequency of a full vacuum, or reduce its cost when we do it? That is, if more transactions are frozen, will a full vacuum be more efficient (primarily in the I/O)? > I believe Fujitsu is looking into what it'd take to make a variant > Postgres with 64-bit XIDs. This'd probably imply also expanding CIDs, > OIDs, and some other things, so the space penalty is not to be sneezed > at ... but it might be worth it for installations like yours. Thanks for the pointer, I'll take a look. --Ian ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [ADMIN] VACUUM and read-mostly tables
On Tue, 2005-04-05 at 11:34, Tom Lane wrote: > Ian Westmacott <[EMAIL PROTECTED]> writes: > > But the question is whether vacuum freezing tables will > > help me reduce the frequency of a full vacuum, or reduce > > its cost when we do it? That is, if more transactions > > are frozen, will a full vacuum be more efficient > > (primarily in the I/O)? > > I'm not sure if you are actually confused, or are just using confusing > terminology ... but there's a significant difference between VACUUM FULL > (ie, vacuum and try to reclaim space) and a database-wide vacuum. I > *think* you are using "full vacuum" to mean "database-wide vacuum" but > it's not entirely clear. Yes, sorry, bad choice of words. By "full vacuum" I meant "database-wide". > Anyway, the frequency with which you have to do database-wide vacuums to > avoid XID wraparound is determined entirely by the rate at which you use > up XIDs. Doing piecemeal VACUUM FREEZEs would reduce the amount of work > that needs to be done in the eventual database-wide vacuum, but it's not > clear that it'd be a net win given the added work of the extra VACUUM > scans. But potentially I could spread the cost out over time, making it less disruptive when it occurs, right? > Have you looked at whether you can slow down the rate of XID consumption > (ie, by bundling operations into larger transactions)? That might be a > more useful route to limiting the costs involved. Yes, I'd like to explore that. Essentially what happens now is that a number of rows are written to each of about two dozen tables, each with a COPY FROM STDIN, and then there are a few INSERTS and UPDATES to boot. As far as we are concerned, all of this could be a single transaction. My understanding is that all the inserts resulting from a COPY are a single transaction, but is it possible to make multiple COPYs a single transaction? Or would I have to do them all as individual INSERTs and make the whole thing a single transaction? Would that be more costly? Thanks, --Ian ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [ADMIN] VACUUM and read-mostly tables
On Tue, 2005-04-05 at 11:39, Jim C. Nasby wrote: > On Tue, Apr 05, 2005 at 11:13:06AM -0400, Ian Westmacott wrote: > > On Tue, 2005-04-05 at 00:41, Jim C. Nasby wrote: > > > We'll only answer if you do a write-up on your database. :P > > > > > > Seriously, those are some seriously big numbers. What else is the > > > database doing? What hardware is it running on? > > > > > > We run on a dual 3.2GHz P4 with 2GB RAM, but are still > > finalizing the storage hardware. We've tried various > > flavors of RAID, filesystems and volume management (and > > are anxious to try out tablespaces in 8). We've found > > fragmentation to be our largest limiting factor. XFS > > helps with that, and seems to provide the highest > > sustained throughput on raw tables, but its not the end > > of the story since fragmentation is still high. > > What else is the database doing besides the inserts? A proportionally small number of updates, no deletes, and a set of moderately complex queries. > And if UFS is available for linux you should might try it. Would UFS help the fragmentation issue? We have seen ext3 allocating blocks in 2-4 pages, while XFS manages 8-16 pages. Thanks, --Ian ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
[ADMIN] database corruption
For several weeks now we have been experiencing fairly severe database corruption upon clean reboot. It is very repeatable, and the corruption is of the following forms: ERROR: could not access status of transaction foo DETAIL: could not open file "bar": No such file or directory ERROR: invalid page header in block foo of relation "bar" ERROR: uninitialized page in block foo of relation "bar" At first, we believed this was related to XFS, and have been pursuing investigations along those lines. However, we have now experienced the exact same problem with JFS. Here are some details: - Postgres 7.4.2 - 2.6.6 kernel.org kernel - dedicated database partition - repeatable with XFS and JFS (have not seen on ext3) - repeatable with and without Linux software RAID 0 - repeatable with IDE and SATA - repeatable with and without fsync, and with fdatasync - repeatable on multiple systems I have two questions: - any known reason why this might be occurring? (we must have something wrong, for this high rate of severe error). - if I don't care about losing data, and am not interested in trying to recover anything, how can I arrange for Postgres to proceed normally? I know about zero_damaged_pages, but this doesn't help with missing transaction files and such. Is there any way to get Postgres to chuck anything bad and proceed? Thanks, --Ian ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [ADMIN] database corruption
Hi Chris, > I think it is important to figure out why this is happening. I would > not want to run any production databases on systems that were failing > like this. You and me both :) (in our application though, it is not a total disaster to lose the last 5 minutes of transactions, it is a disaster if the database is unusable when it comes up) > 1) Any other computers suffer random application crashes, power downs, > etc. in your building? No, but more importantly, we have seen this failure happen in different buildings (in different cities), on same spec but different hardware (at least three motherboards and power supplies, six disks). That's why it really feels like a bug or configuration error. > 2) I take it there are no Raid controllers involved? No. But we get this error with and without software RAID, FWIW. > 3) RAM is non-ECC? I'll have to double-check, but I think it is. > 4) Are the systems on UPS's? Yes. > If I could make a wild (and probably wrong) guess, I would wonder if > something external to the system (like electrical supply) was > introducing glitches into memory, causing bad data to be written. I am > only mentioning it because I have implicated electrical supply in other > cases where rare computer failurres weer affecting many systems... I would tend to agree, but this occurs on multiple systems in multiple locations (but, oddly enough, we are having trouble reproducing it in our lab). And we have run memtest. However, it is true that all the systems on which this has been seen have the same spec power supply/UPS. I would think though, that this could cause error at any time -- all of these failures occur after reboot (that is, no corruption, reboot, immediate corruption). I have stopped/started Postgres while the application is running, without corruption. (smells like a dirty buffer not being written to disk, which is why we focused on the filesystem). Here are some further details: - 865PE/G Neo2-P (MS-6728) ATX motherboard (and similar for IDE) - 2x 512MB/400MHz DIMM RAM - Intel Pentium 4/3.2GHz/1MB/800MHz CPU (hyperthreading enabled) - 2x WD 250GB/7200RPM/8MB/SATA-150 on ICH5 SATA ports (also tested similar IDE drives), writethrough - XFS and JFS (not seen on ext3, but not fully tested) - either software RAID 0 on both drives, or one drive alone without RAID - SuSE 9.1 - 2.6.6 kernel - Postgres 7.4.2 - 300 TPS against DB containing 5-50GB data, no more than a dozen concurrent connections. - fsync (or not) and fdatasync - Postgres may be taken down (via init script) with connections open to it (in fact the application may aggressively try to re-establish the connection as it goes down). - we have put syncs, sleeps and large dd to the disk in the shutdown scripts, none of which work. At this point, I'm really looking for fresh ideas. Thanks, --Ian ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [ADMIN] database corruption
Thanks for the tip. Later kernel versions have unrelated problems for us, but we'll take a look at the filesystem mods and see if we can backpatch them. --Ian > I remember a problem that was fixed in the 2.6.9 kernel concerning XFS > corruption (shutdowns I think were the worst). Also introduced some JFS > stuff, but I don't run JFS so I didn't really pay much attention to > that. > > Can you try a later (the latest?) vanilla kernel? ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [ADMIN] database corruption
> In your previous emails, you stated that these errors were seen on > multiple systems. Multiple systems, configured identically, with > diverse motherboards/hardware or always identical hardware except for > the sata/IDE drives? > > I ask, because I noted that you are using the Neo 2 ATX motherboard that > has the "dynamic overclocking" feature. This can cause grave > instability in systems when it set to an aggressive setting. This > motherboard is not compatible with low quality ram either. This > motherboard also is prone to heat problems, and requires extra cooling. > If you have multiple systems failing, using this motherboard, then that > particular model could be the culprit. We have seen this using at least one other motherboard besides the Neo 2 ATX. And (fortunately or unfortunately) we don't use the dynamic overclocking feature. Do you know how the RAM incompatibility is manifest? Thanks, --Ian ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
[ADMIN] 8.1.8 autovacuum missing databases
I have a Postgres 8.1.8 system with three databases. Although autovacuum is enabled (I'm using all default autovacuum configuration settings) and pg_autovacuum is empty, the logs indicate that only template1 and one of the three databases are being processed by autovacuum. Are there other reasons why the other two databases might be skipped, or some way I could diagnose the issue? Thanks, --Ian -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
Re: [ADMIN] 8.1.8 autovacuum missing databases
On Tue, 2008-04-29 at 16:33 -0400, Alvaro Herrera wrote: > Is autovacuum dying before being able to finish the vacuuming of > template1 or the other database? Not as far as I can tell. There are no indications of any crash or error in the log file (I just bumped the log level up to debug1). Just autovacuum processing template1 or the one other database every minute. I ran VACUUM ANALYZE on both template1 and the one database manually just to make sure there wasn't a lock or something preventing completion, but they both completed without error. --Ian -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
Re: [ADMIN] 8.1.8 autovacuum missing databases
On Tue, 2008-04-29 at 17:12 -0400, Alvaro Herrera wrote: > Can you verify whether age(pg_database.datfrozenxid) is shrinking after > vacuuming? If the age() of a database is higher than the applicable > max_freeze_age, then it will always be chosen. One of the databases is about 1.5TB, so that could take a while. Anything else I could look at in the meantime? The age of these databases right now is about 290039577. Also, autovacuum doesn't always choose the same DB, it alternates between template1 and one of mine -- going back through a couple of months of logs I don't see it ever choosing the same one twice in a row. I'll note that I have 5 of these essentially identical systems (same hardware/software platform, databases, configuration, and load). Two of them have this issue (autovacuum processing multiple databases, but not all) and the other three don't. A commonality between the two that have this issue is that template1 is processed but postgres is not. One of the two is processing the 1.5TB DB but the other is not. Thanks, --Ian -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
Re: [ADMIN] 8.1.8 autovacuum missing databases
On Wed, 2008-04-30 at 12:27 -0400, Tom Lane wrote: > Alvaro Herrera <[EMAIL PROTECTED]> writes: > > I still think that the autovac is dying before completing the task. Did > > you investigate whether there are "ERROR" messages coming from > > autovacuum? No PG crashes would happen. There are no ERROR messages at all. Sample log snippet: 2008-04-30 12:03:36 EDT LOG: autovacuum: processing database "template1" 2008-04-30 12:04:36 EDT LOG: autovacuum: processing database "itvtrackdatapos" 2008-04-30 12:05:36 EDT LOG: autovacuum: processing database "template1" 2008-04-30 12:06:36 EDT LOG: autovacuum: processing database "itvtrackdatapos" 2008-04-30 12:07:36 EDT LOG: autovacuum: processing database "template1" 2008-04-30 12:08:36 EDT LOG: autovacuum: processing database "itvtrackdatapos" 2008-04-30 12:09:36 EDT LOG: autovacuum: processing database "template1" 2008-04-30 12:10:36 EDT LOG: autovacuum: processing database "itvtrackdatapos" > I'm noticing though that the 8.1 logic pays attention to both > datvacuumxid and datfrozenxid. Could we see the age() of both of > those columns for all the databases. The are all identical: itvtrackdata=> select datname,age(datfrozenxid),age(datvacuumxid) from pg_database; datname |age|age --+---+--- postgres | 295995059 | 295995059 itvtrackdata | 295995059 | 295995059 itvtrackdatauser | 295995059 | 295995059 itvtrackdatapos | 295995059 | 295995059 template1| 295995059 | 295995059 template0| 295995059 | 295995059 (6 rows) itvtrackdata=> --Ian -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
Re: [ADMIN] 8.1.8 autovacuum missing databases
On Wed, 2008-04-30 at 13:07 -0400, Tom Lane wrote:
> Oh, that's really strange. Could we see
>
> select ctid,xmin,* from pg_database
itvtrackdata=> select ctid,xmin,* from pg_database;
ctid | xmin | datname | datdba | encoding | datistemplate |
datallowconn | datconnlimit | datlastsysoid | datvacuumxid | datfrozenxid |
dattablespace | datconfig | datacl
---+--+--++--+---+--+--+---+--+--+---+---+
(0,1) | 564 | postgres | 10 |6 | f | t
| -1 | 10792 | 499 | 499 |
1663 | |
(0,2) | 577 | itvtrackdata | 16384 |6 | f | t
| -1 | 10792 | 499 | 499 |
1663 | |
(0,3) | 605 | itvtrackdatauser | 16384 |6 | f | t
| -1 | 10792 | 499 | 499 |
1663 | |
(0,5) | 18350176 | itvtrackdatapos | 16384 |6 | f | t
| -1 | 10792 | 499 | 499 |
1663 | |
(0,6) | 557 | template1| 10 |6 | t | t
| -1 | 10792 | 499 | 499 |
1663 | | {postgres=CT/postgres}
(0,7) | 558 | template0| 10 |6 | t | f
| -1 | 10792 | 499 | 499 |
1663 | | {postgres=CT/postgres}
(6 rows)
itvtrackdata=>
--Ian
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
Re: [ADMIN] 8.1.8 autovacuum missing databases
On Wed, 2008-04-30 at 16:43 -0400, Tom Lane wrote: > Ian Westmacott <[EMAIL PROTECTED]> writes: > 499 is the value that those columns would have immediately after initdb, > in an 8.1 database. So what this says is that vacuum has never > succeeded in updating the values at all, in any of your databases. > It definitely *should* be doing that given the size of the age() values > you're reporting. Moreover, after a look through the 8.1.8 source code > I cannot see how it would not update the values without throwing an > ERROR or at least a WARNING into the postmaster log. (What have you got > log_min_messages set to, anyway? Maybe the complaint is getting > suppressed?) log_min_messages is set to the default (notice), until I reset it to debug1 a couple of days ago. I combed back through the logs to initial installation in Feb. There are no ERRORs or WARNINGs that are obviously related. However, what I did find was a couple of small log files with timestamps in the future (Jan 2009). Baffled, I did some digging around and it looks like the user initially installed this system with the system clock set in Jan 2009. The system ran for about 20 minutes, during which time two of our application databases were created and processed by autovacuum. Then the user corrected the time problem. During that 20 minutes in the future, autovacuum processed postgres, template1, and two of our databases (itvtrackdata and itvtrackdatauser). autovacuum hasn't touched our two since time was corrected. I imagine if a database's last process time is in the future, that would mess up autovacuum's choice algorithm (at the least). Can I confirm this is the problem? Is there a way to fix it short of dump/restore? Anything else I should worry about in this installation? I mentioned earlier that I had two installations like this. The second one doesn't seem to have any time issues. But in that case the only database being skipped is postgres. Do I need to worry about that? Thanks, --Ian -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
Re: [ADMIN] 8.1.8 autovacuum missing databases
On Thu, 2008-05-01 at 09:57 -0400, Tom Lane wrote: > AFAICS there is no convenient way in 8.1 to examine the per-database > last_autovac_time entries, which'd be needed to confirm this theory. > We should check that just to be sure. Please do the following: > > 1. Stop the postmaster. > 2. Move the file $PGDATA/global/pgstat.stat somewhere else. > 3. Restart the postmaster. > 4. Send pgstat.stat as a binary attachment to me or Alvaro. Done; I'll send pgstat.stat separately. > This will reset all your statistics counters and also the > last_autovac_time settings. If autovac starts to behave more > normally then we'll know that was it. I don't know if it is normal, but it is different. Now its just going back and forth between itvtrackdata and itvtrackdatapos: 2008-05-01 13:30:37 EDT 10482 LOG: transaction ID wrap limit is 2147484146, limited by database "postgres" 2008-05-01 13:32:03 EDT 11295 LOG: autovacuum: processing database "itvtrackdata" 2008-05-01 13:34:03 EDT 12384 LOG: autovacuum: processing database "itvtrackdatapos" 2008-05-01 13:35:03 EDT 12897 LOG: autovacuum: processing database "itvtrackdata" 2008-05-01 13:36:33 EDT 13769 LOG: autovacuum: processing database "itvtrackdatapos" 2008-05-01 13:37:33 EDT 14292 LOG: autovacuum: processing database "itvtrackdata" 2008-05-01 13:39:03 EDT 15124 LOG: autovacuum: processing database "itvtrackdatapos" 2008-05-01 13:40:03 EDT 15713 LOG: autovacuum: processing database "itvtrackdata" 2008-05-01 13:41:03 EDT 16387 LOG: autovacuum: processing database "itvtrackdatapos" 2008-05-01 13:42:03 EDT 16925 LOG: autovacuum: processing database "itvtrackdata" 2008-05-01 13:43:03 EDT 17510 LOG: autovacuum: processing database "itvtrackdatapos" 2008-05-01 13:44:03 EDT 18085 LOG: autovacuum: processing database "itvtrackdata" 2008-05-01 13:45:03 EDT 18722 LOG: autovacuum: processing database "itvtrackdatapos" 2008-05-01 13:46:03 EDT 19447 LOG: autovacuum: processing database "itvtrackdata" > > I mentioned earlier that I had two installations like this. The second > > one doesn't seem to have any time issues. But in that case the only > > database being skipped is postgres. Do I need to worry about that? > > Is the age() getting unreasonably large for postgres? It might be > skipping it because there's been no activity. itvtrackdata=> select datname,age(datfrozenxid),age(datvacuumxid) from pg_database; datname |age|age --+---+--- postgres | 157900061 | 157900061 itvtrackdata | 157900061 | 157900061 itvtrackdatauser | 157900061 | 157900061 itvtrackdatapos | 157900061 | 157900061 template1| 157900061 | 157900061 template0| 157900061 | 157900061 (6 rows) itvtrackdata=> (they're all 499 too). Thanks, --Ian -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
