Re: [HACKERS] Kudos for Reviewers -- straw poll

2013-06-26 Thread Atri Sharma
For me, B,B and another B works. Regards, Atri -- Regards, Atri l'apprenant -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Bloom Filter lookup for hash joins

2013-06-26 Thread Atri Sharma
On Wed, Jun 26, 2013 at 2:09 PM, Pavel Stehule wrote: > Hello > > it looks interesting - please, try to send some performance results, > > Regards > > Pavel Hi, Sure. I will think more about it and put up a design on the list soon. My current aim would be to work on hash joins. If it works well

[HACKERS] Bloom Filter lookup for hash joins

2013-06-25 Thread Atri Sharma
Hi All, I have been researching bloom filters and discussed it on IRC with RhodiumToad and David Fetter, and they pointed me to the various places that could potentially have bloom filters, apart from the places that already have them currently. I have been reading the current implementation of h

Re: [HACKERS] [9.4 CF 1] The Commitfest Slacker List

2013-06-24 Thread Atri Sharma
On Mon, Jun 24, 2013 at 10:54 PM, Josh Berkus wrote: > >>> I will be more than happy to resign as CFM and turn it over to someone >>> else if people have a problem with it. >> >> Heck, Josh. People have to be allowed to critize *a small part* of your >> work without you understanding it as a funda

Re: [HACKERS] Hardware donation

2013-06-21 Thread Atri Sharma
On Fri, Jun 21, 2013 at 10:18 PM, Jim Nasby wrote: > We've got some recently decommissioned servers and Enova is willing to > donate 2 of them to the community. > > There's nothing terribly spectacular about the servers except for memory. We > have one 512G server available and the other would be

Re: [HACKERS] 9.4 CF1 Starts Saturday: need patches, reviewers, asst. CFM

2013-06-11 Thread Atri Sharma
On Tue, Jun 11, 2013 at 11:46 PM, Stephen Frost wrote: > Josh, > > * Josh Berkus (j...@agliodbs.com) wrote: >> We already have 72 patches pending. > > Fun! > >> If you are available to do CF reviews, and want to be assigned a patch >> instead of choosing one for yourself, then please join the >> p

Re: [HACKERS] PostgreSQL Process memory architecture

2013-05-27 Thread Atri Sharma
>We may still be able to do better than what we're doing > today, but I'm still suspicious that you're going to run into other > issues with having 500 indexes on a table anyway. +1. I am suspicious that the large number of indexes is the problem here,even if the problem is not with book keeping

Re: [HACKERS] PostgreSQL Process memory architecture

2013-05-27 Thread Atri Sharma
On Mon, May 27, 2013 at 6:02 PM, Ben Zeev, Lior wrote: > Hi Stephen, > > The case which I'm seeing is that I have an empty table without any rows, > Create table test ( > Num Integer, > C1 character varying(512), > C2 character varying(512), > C3 character varying(512)); > > I create sever

Re: [HACKERS] PostgreSQL Process memory architecture

2013-05-27 Thread Atri Sharma
> I'd expect the performance issue would be from planner time more than > memory usage- but if there is a serious memory usage issue here, then > it'd be valuable to have a test case showing what's happening. We may > not be releasing the sys cache in some cases or otherwise have a bug in > this a

Re: [HACKERS] PostgreSQL Process memory architecture

2013-05-27 Thread Atri Sharma
> This is not generally a reason to avoid indexes. Indexes require more > disk space and must be kept up to date, making them expensive to > maintain due to increased disk i/o. Building an index uses as much > memory as it's allowed to- it uses maintenance_work_mem to limit itself. Yes, too many

Re: [HACKERS] PostgreSQL Process memory architecture

2013-05-27 Thread Atri Sharma
> A better place to look would be the documentation for the release of PG > which you are on, or the latest release otherwise, which is: > > http://www.postgresql.org/docs/9.2/static/storage.html Oops,yes,sorry about that. Thanks a ton for pointing that out. Regards, Atri -- Regards, Atri l'

Re: [HACKERS] PostgreSQL Process memory architecture

2013-05-27 Thread Atri Sharma
>> AFAIK, the shared disk buffers are the only part shared between the >> processes. > > There's a bit of other information shared, but disk buffers are > certainly the bulk of it. The other information being locks? Regards, Atri -- Regards, Atri l'apprenant -- Sent via pgsql-hackers maili

Re: [HACKERS] PostgreSQL Process memory architecture

2013-05-27 Thread Atri Sharma
> An index is built in backend process's local memory, but, when > accessing, index pages are stored in shared memory. That is, for > example, when an index scan is performed, index pages are brought into > shared memory and accessed from there. > > Yes, brought into the shared disk buffers and re

Re: [HACKERS] PostgreSQL Process memory architecture

2013-05-27 Thread Atri Sharma
On Mon, May 27, 2013 at 3:55 PM, Ben Zeev, Lior wrote: > Thanks Atri! > > Do you know why PostgreSQL store the indexes in memory per process and not in > the shared memory? > Is there a way to prevent it store the indexes data per process, and force it > storing it in the shared memory? Ok, so

Re: [HACKERS] PostgreSQL Process memory architecture

2013-05-27 Thread Atri Sharma
On Mon, May 27, 2013 at 3:41 PM, Ben Zeev, Lior wrote: > Hi Atri, > > Thanks for your answer! > Do you have idea what may be the reason that PostgreSQL process consume more > memory when there are more partial indexes on the DB table? Well, I am not too sure, but indexes always take up more spa

Re: [HACKERS] PostgreSQL Process memory architecture

2013-05-27 Thread Atri Sharma
> Does each PostgreSQL process allocating in its own memory (Not shared > memory) a cache of all the database catalog which it access during the SQL > execution? > > I mean does each process holds all the catalog indexes data which it > accessed, all the catalog index statistics etc’ accessed AFAI

Re: [HACKERS] Time limit for a process to hold Content lock in Buffer Cache

2013-05-23 Thread Atri Sharma
> Right, the actual signal handler will only kill the query immediately > if the backend is in a safe state (eg, while it's waiting for a > heavyweight lock). Otherwise it just sets a flag that's checked by > CHECK_FOR_INTERRUPTS. See StatementCancelHandler in postgres.c. Roger that, I will defi

Re: [HACKERS] Time limit for a process to hold Content lock in Buffer Cache

2013-05-23 Thread Atri Sharma
> > A little bit --- the timeout won't actually kill the query until the > next time control reaches a CHECK_FOR_INTERRUPTS macro that's not inside > a critical section. We've had issues in the past with particular code > paths that failed to include such a check in a long-running loop, and > ther

Re: [HACKERS] Time limit for a process to hold Content lock in Buffer Cache

2013-05-23 Thread Atri Sharma
On Thu, May 23, 2013 at 8:52 PM, Amit Langote wrote: >> >> If you let an uncooperative user issue arbitrary SQL queries, he can >> do any number of things to put server performance into the tank. >> For instance, take out exclusive locks on all your tables and just >> go to sleep (although I think

Re: [HACKERS] Time limit for a process to hold Content lock in Buffer Cache

2013-05-23 Thread Atri Sharma
> No, you're not following. These would be table-level heavyweight locks, > and there already are timeout mechanisms that work at that level. > Oh,right.Sorry,I missed that. I will go and research heavyweight locks now. Thanks a ton! Regards, Atri -- Regards, Atri l'apprenant -- Sent via

Re: [HACKERS] Time limit for a process to hold Content lock in Buffer Cache

2013-05-23 Thread Atri Sharma
> For instance, take out exclusive locks on all your tables and just > go to sleep (although I think this is limited by table permissions in > recent PG versions). This is what I have been worried about. The locks(exclusive locks or content locks in buffers) can be held and stalled. Cant we do any

Re: [HACKERS] Time limit for a process to hold Content lock in Buffer Cache

2013-05-23 Thread Atri Sharma
On Thu, May 23, 2013 at 8:18 PM, Tom Lane wrote: > Atri Sharma writes: >> I was musing over a possible condition where a rogue client gets the >> backend to process queries which take a *lot* of time(note, this is >> only in my head atm.I may be completely wrong here). >

Re: [HACKERS] Time limit for a process to hold Content lock in Buffer Cache

2013-05-23 Thread Atri Sharma
On Thu, May 23, 2013 at 8:01 PM, Tom Lane wrote: > Atri Sharma writes: >> On a different note, shouldn't we have a time out for a content lock >> in buffer cache? > > No; the overhead of setting up and canceling such a timeout would > greatly outweigh any possible be

Re: [HACKERS] Time limit for a process to hold Content lock in Buffer Cache

2013-05-23 Thread Atri Sharma
Sent from my iPad On 23-May-2013, at 16:27, Amit Kapila wrote: > On Thursday, May 23, 2013 3:35 PM Atri Sharma wrote: >> Hi all, >> >> I was searching for an existing mechanism which ensures that a crashed >> process releases a content lock held on a page in buffe

[HACKERS] Time limit for a process to hold Content lock in Buffer Cache

2013-05-23 Thread Atri Sharma
Hi all, I was searching for an existing mechanism which ensures that a crashed process releases a content lock held on a page in buffer cache. Also, a similar mechanism for refcount of the page. I seem to have missed it. Do we have a mechanism,and if yes,where can I find it? Regards, Atri -- R

Re: [HACKERS] Why there is a union in HeapTupleHeaderData struct

2013-05-20 Thread Atri Sharma
On Mon, May 20, 2013 at 8:54 PM, Alvaro Herrera wrote: > Hello, > > Would you guys please trim the quoted text of the emails you're replying > to? I understand Gmail is obnoxious w.r.t. quoted text, but this is > starting to become excessive. Oops, I didnt notice that. Sorry! Regards, Atri -

Re: [HACKERS] Why there is a union in HeapTupleHeaderData struct

2013-05-20 Thread Atri Sharma
Sent from my iPad On 20-May-2013, at 18:14, Amit Langote wrote: > Wonder though if this question is better asked in pgsql-novice? > > On Mon, May 20, 2013 at 9:23 PM, Amit Langote wrote: >> Hello, >> >> I think a more appropriate question to be asked here would be at what >> point (in the l

Re: [HACKERS] Road map to study about fetching a set of tuples - novice!

2013-05-18 Thread Atri Sharma
Sent from my iPad On 18-May-2013, at 20:49, "Dickson S. Guedes" wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Em 18-05-2013 11:40, Atri Sharma escreveu: >> On 18-May-2013, at 20:01, Soroosh Sardari >> wrote: >> >>> Hi >>

Re: [HACKERS] Road map to study about fetching a set of tuples - novice!

2013-05-18 Thread Atri Sharma
Sent from my iPad On 18-May-2013, at 20:01, Soroosh Sardari wrote: > Hi > > I was tracing a simple SELECT query to find how pg works for fetching tuples. > but I'm totally lost in the code. > Could you help me to understand under the hood? > I know about parsing and planning parts, my actual

Re: [HACKERS] request a new feature in fuzzystrmatch

2013-05-18 Thread Atri Sharma
On Sat, May 18, 2013 at 1:47 PM, Liming Hu wrote: > On 5/18/2013 1:03 AM, Atri Sharma wrote: >> >> On Sat, May 18, 2013 at 1:30 PM, Liming Hu wrote: >>> >>> On 5/18/2013 12:11 AM, Atri Sharma wrote: >>> >>> Sent from my iPad >>> >>

Re: [HACKERS] request a new feature in fuzzystrmatch

2013-05-18 Thread Atri Sharma
On Sat, May 18, 2013 at 1:30 PM, Liming Hu wrote: > On 5/18/2013 12:11 AM, Atri Sharma wrote: > > Sent from my iPad > > On 18-May-2013, at 12:34, Liming Hu wrote: > > On Fri, May 17, 2013 at 11:44 PM, Atri Sharma wrote: > > Sent from my iPad > > On 18-May

Re: [HACKERS] request a new feature in fuzzystrmatch

2013-05-18 Thread Atri Sharma
Sent from my iPad On 18-May-2013, at 12:34, Liming Hu wrote: > On Fri, May 17, 2013 at 11:44 PM, Atri Sharma wrote: >> >> >> Sent from my iPad >> >> On 18-May-2013, at 12:07, Liming Hu wrote: >> >>> On Fri, May 17, 2013 at 11:13 PM, Atri S

Re: [HACKERS] request a new feature in fuzzystrmatch

2013-05-17 Thread Atri Sharma
Sent from my iPad On 18-May-2013, at 12:07, Liming Hu wrote: > On Fri, May 17, 2013 at 11:13 PM, Atri Sharma wrote: >> >> >> Sent from my iPad >> >> On 18-May-2013, at 11:23, Liming Hu wrote: >> >>> On Fri, May 17, 2013 at 10:21 PM, Atri S

Re: [HACKERS] request a new feature in fuzzystrmatch

2013-05-17 Thread Atri Sharma
Sent from my iPad On 18-May-2013, at 11:23, Liming Hu wrote: > On Fri, May 17, 2013 at 10:21 PM, Atri Sharma wrote: >> >> >> Sent from my iPad >> >> On 18-May-2013, at 10:41, Liming Hu wrote: >> >>> On Fri, May 17, 2013 at 9:39 PM, Atri S

Re: [HACKERS] request a new feature in fuzzystrmatch

2013-05-17 Thread Atri Sharma
Sent from my iPad On 18-May-2013, at 10:41, Liming Hu wrote: > On Fri, May 17, 2013 at 9:39 PM, Atri Sharma wrote: >> >> >> Sent from my iPad >> >> On 18-May-2013, at 10:00, Liming Hu wrote: >> >>> On 5/17/2013 9:10 PM, Atri Sharma wrote:

Re: [HACKERS] request a new feature in fuzzystrmatch

2013-05-17 Thread Atri Sharma
Sent from my iPad On 18-May-2013, at 10:00, Liming Hu wrote: > On 5/17/2013 9:10 PM, Atri Sharma wrote: >> On Sat, May 18, 2013 at 9:27 AM, Liming Hu wrote: >>> On 5/17/2013 12:39 PM, Liming Hu wrote: >>>> On Fri, May 17, 2013 at 12:06 PM, Josh Berkus wrote:

Re: [HACKERS] request a new feature in fuzzystrmatch

2013-05-17 Thread Atri Sharma
On Sat, May 18, 2013 at 9:27 AM, Liming Hu wrote: > On 5/17/2013 12:39 PM, Liming Hu wrote: >> >> On Fri, May 17, 2013 at 12:06 PM, Josh Berkus wrote: Can we add Levenshtein-Damerau edit distance since it is used in spelling checker instead of levenshtein edit distance? >>> >>

Re: [HACKERS] request a new feature in fuzzystrmatch

2013-05-17 Thread Atri Sharma
On Fri, May 17, 2013 at 9:29 PM, Alvaro Herrera wrote: > Liming Hu escribió: >> Hi Atri, >> >> I also talked to the author of fuzzystrmatch, he also thinks it is a >> good feature. > > Yeah, I heard our fuzzystrmatch module being talked down a decade ago > for not considering transpositions, so +1

Re: [HACKERS] request a new feature in fuzzystrmatch

2013-05-17 Thread Atri Sharma
On Fri, May 17, 2013 at 8:56 PM, Liming Hu wrote: > On 5/17/2013 8:23 AM, Atri Sharma wrote: >> >> On Fri, May 17, 2013 at 8:51 PM, Liming Hu wrote: >>> >>> Hi Atri, >>> >>> Thanks for the quick response. >>> >>>

Re: [HACKERS] request a new feature in fuzzystrmatch

2013-05-17 Thread Atri Sharma
On Fri, May 17, 2013 at 8:51 PM, Liming Hu wrote: > Hi Atri, > > Thanks for the quick response. > > levenshtein edit distance defines operations of: insertion, deletion, > modification. > Levenshtein-Damerau edit distance defines operations of: insertion, > deletion, modification > and transposit

Re: [HACKERS] request a new feature in fuzzystrmatch

2013-05-17 Thread Atri Sharma
Sent from my iPad On 17-May-2013, at 6:11, Liming Hu wrote: > Hi, > > I am studying your levenshtein edit distance contribution in Postgresql. > > Can we add Levenshtein-Damerau edit distance since it is used in > spelling checker > instead of levenshtein edit distance? > > Levenshtein-Dame

Re: [HACKERS] postgres_fdw foreign tables and serial columns

2013-05-15 Thread Atri Sharma
Sent from my iPad On 15-May-2013, at 18:56, Tom Lane wrote: > "Nicholson, Brad (Toronto, ON, CA)" writes: >> [ this error message sucks: ] >> test=# create foreign table local_foo (id serial) server test_server options >> (table_name 'foo');

Re: [HACKERS] Graph datatype addition

2013-05-08 Thread Atri Sharma
> > Your second drawing didn't really make any sense to me. :( > > I do think it would be most productive to focus on what the API for dealing > with graph data would look like before trying to handle the storage aspect. > The storage is potentially dirt-simple, as others have shown. The only > cha

Re: [HACKERS] Graph datatype addition

2013-05-08 Thread Atri Sharma
On Thu, May 2, 2013 at 7:58 AM, Atri Sharma wrote: > > > Sent from my iPad > > On 02-May-2013, at 4:33, Misa Simic wrote: > > > > On Wednesday, May 1, 2013, Atri Sharma wrote: >> >> Hi all, >> >> Please find a probable prototype for the s

Re: [HACKERS] GSOC13 proposal - extend RETURNING syntax

2013-05-02 Thread Atri Sharma
Sent from my iPad On 03-May-2013, at 0:07, David Fetter wrote: > On Thu, May 02, 2013 at 01:40:59PM -0400, Tom Lane wrote: >> David Fetter writes: >>> On Thu, May 02, 2013 at 06:28:53PM +0200, Andres Freund wrote: prior/after? Both are unreserved keywords atm and it seems far less l

Re: [HACKERS] Graph datatype addition

2013-05-01 Thread Atri Sharma
Sent from my iPad On 02-May-2013, at 4:33, Misa Simic wrote: > > > On Wednesday, May 1, 2013, Atri Sharma wrote: >> Hi all, >> >> Please find a probable prototype for the same: >> >> struct GraphNode >> { >> Oid NodeOid;// Oid of

Re: [HACKERS] Graph datatype addition

2013-05-01 Thread Atri Sharma
; On Mon, Apr 29, 2013 at 10:51 PM, Peter Eisentraut wrote: >> On Sun, 2013-04-28 at 22:55 -0700, Atri Sharma wrote: >>> If we add support for weighted graphs, we can probably add support for >>> some common graph algorithms, such as Djikstra's algorithm, Bellman

Re: [HACKERS] Graph datatype addition

2013-04-30 Thread Atri Sharma
> actually, i was going to suggest to Atri to take a look at that, > pgrouting is currently in develop of v2.0 which will rewrite some > parts (including some of the algorithms). > > Maybe this datatype could fit better as part of pgrouting Thanks, I will have a look at pgRouting.Although, in a sh

Re: [HACKERS] Graph datatype addition

2013-04-30 Thread Atri Sharma
> 1. We created JSON and XML types as ways of storing data that has a robust > validation system. > > They're still, in a sense, just "plain old text", but it's "plain old text" > that the user can be certain satisfies the respective rules for > representations. Yes, although, I feel that we can

Re: [HACKERS] Graph datatype addition

2013-04-30 Thread Atri Sharma
> I believe it would be best to largely separate the questions of storage and > access. Partly because of Florian's concern that you'd frequently want only > one representation of the whole graph, but also because the actual storage > interface does NOT have to be user friendly if we have a good ac

Re: [HACKERS] Graph datatype addition

2013-04-30 Thread Atri Sharma
> Have you considered maybe ltree datatype? > > To me all described sounds solveable on pure sql way ( + ltree datatype to > help with indexes and performance as materialised path to avoid recursive > query all the time...) This may build over the existing solutions itself. I will have to check th

Re: [HACKERS] Graph datatype addition

2013-04-30 Thread Atri Sharma
> Usually though, you'd be interested a large graphs which include > information for lots of records (e.g., nodes are individual users, > or products, or whatever). A graph datatype is not well suited for > that, because it'd store each graph as a single value, and updating > the graph would mean r

Re: [HACKERS] Graph datatype addition

2013-04-29 Thread Atri Sharma
On Mon, Apr 29, 2013 at 10:12 PM, Misa Simic wrote: > Hi Atri, > > What is an example of custom internal representation and its JSON > representation (though and JSON and HStore represent its value as text)? > > I also think that the key question is: "what operations would you support > on this

Re: [HACKERS] Graph datatype addition

2013-04-29 Thread Atri Sharma
> > I don't agree with this; JSON is not really designed to store graphs. > You will probably need a customized internal representation, just like > hstore, that expresses a graph like structure. Yes, we will have a custom internal representation. I was thinking of ways to export the graph into us

Re: [HACKERS] Graph datatype addition

2013-04-29 Thread Atri Sharma
> > This is an interesting idea. Historically I've always decomposed > graphs into relational structures because that's the only practical > way to query them. Graphs are not currently able to be transported > out of the database currently via JSON so one of the areas to focus > your research wi

Re: [HACKERS] Graph datatype addition

2013-04-28 Thread Atri Sharma
> It's probably pretty easy to add this, but I think the question is > what would make it better than storing the same representation in a > text field. I completely agree. The main point in making a new datatype would be to add support for operations that are normally done with graphs. >Obvious

[HACKERS] Graph datatype addition

2013-04-27 Thread Atri Sharma
Hi all, Inspired by the awesome work done by Oleg sir in HStore, I have been thinking about making a graph data type as an extension in postgres. I have been reading about graph databases and how storing data in graphs can lead to some really awesome functionalities such as social network analy

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-04-02 Thread Atri Sharma
Sent from my iPad On 02-Apr-2013, at 23:41, Merlin Moncure wrote: > On Tue, Apr 2, 2013 at 12:50 PM, Atri Sharma wrote: >> On Tue, Apr 2, 2013 at 9:24 PM, Robert Haas wrote: >> >>> One thought I had for fiddling with usage_count is to make it grow >>>

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-04-02 Thread Atri Sharma
On Tue, Apr 2, 2013 at 9:24 PM, Robert Haas wrote: > One thought I had for fiddling with usage_count is to make it grow > additively (x = x + 1) and decay exponentially (x = x >> 1). I'm not > sure the idea is any good, but one problem with the current system is > that it's pretty trivial for a

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-04-01 Thread Atri Sharma
> I don't be believe there is any reasonable argument that > sitting and spinning while holding the BufFreelistLock is a good idea. I completely agree. The idea of spinning for a lock while already inside a lock seems like a source of a hit to performance. Regards, Atri -- Regards, Atri l'

Re: [HACKERS] GSoC project : K-medoids clustering in Madlib

2013-03-27 Thread Atri Sharma
> But it would be even better if MADLib would apply to GSoC as an independent > organization. The deadline for organization applications is on March 29th, > so if the MADLIb people are interested in that, they need to hurry and send > the application right now. Agreed. Is there any way we could ad

Re: [HACKERS] GSoC project : K-medoids clustering in Madlib

2013-03-26 Thread Atri Sharma
>> I'm a bit confused as to why this is being proposed as a >> Postgres-related project. I don't even know what MADlib is, but I'm >> pretty darn sure that no part of Postgres uses it. KNNGist certainly >> doesn't. > > It's a reasonably well established extension for Postgres for > statistical an

Re: [HACKERS] GSoC project : K-medoids clustering in Madlib

2013-03-26 Thread Atri Sharma
I suggested a couple of algorithms to be implemented in MADLib(apart from K Medoids). You could pick some(or all) of them, which would require 3 months to be completed. As for more information on index, you can refer http://wiki.postgresql.org/wiki/What's_new_in_PostgreSQL_9.1 along with the pos

Re: [HACKERS] Limiting setting of hint bits by read-only queries; vacuum_delay

2013-03-25 Thread Atri Sharma
> This is pretty similar to the proposal Atri and I just recently made. > I am 100% in agreement that something must be done here...SELECT has > none of the i/o mitigation features that vacuum has. Is your idea > better? probably (although you have to give a small penalty for a user > facing tunab

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-24 Thread Atri Sharma
>If we > use the value calculated locally to decide on eviction, the highly > used buffers where this is likely will get at least one clock sweep > cycle of grace time. If they are indeed highly used it's likely that > someone will manage to bump the usage_count in the meanwhile. If they > are not

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-24 Thread Atri Sharma
On Sun, Mar 24, 2013 at 6:11 AM, Greg Smith wrote: > On 3/22/13 8:45 AM, Ants Aasma wrote: >> >> However, I think the main issue isn't finding new algorithms that are >> better in some specific circumstances. The hard part is figuring out >> whether their performance is better in general. > > > Ri

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-24 Thread Atri Sharma
> > I'll have to take a look. Removing *all spinning* from from page > allocation though feels like it might be worthwhile to test (got to > give some bonus points for being a very local change and simple to > implement). I wonder if with more shared buffers you tend to sweep > more buffers per a

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-24 Thread Atri Sharma
> Perhaps this isn't the help you were looking for, but I spent a long time > looking into this a few years ago. Then I stopped and decided to work on > other things. I would recommend you do so too. Agreed. It seems that my concerns were not valid, and since you have already done some testing h

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-23 Thread Atri Sharma
> > > Partitioned clock sweep strikes me as a bad idea... you could certainly get > unlucky and end up with a lot of hot stuff in one partition. > > Another idea that'sbeen broughht up inthe past is to have something in the > background keep a minimum number of buffers on the free list. That's how

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-22 Thread Atri Sharma
> > Moreover, if the buffer happens to miss a decrement due to a data > race, there's a good chance that the buffer is heavily used and > wouldn't need to be evicted soon anyway. (if you arrange it to be a > read-test-inc/dec-store operation then you will never go out of > bounds) However, clockswe

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-22 Thread Atri Sharma
On Fri, Mar 22, 2013 at 11:36 PM, Greg Stark wrote: > On Fri, Mar 22, 2013 at 2:02 PM, Tom Lane wrote: >> And we definitely looked at ARC > > We didn't just look at it. At least one release used it. Then patent > issues were raised (and I think the implementation had some contention > problems).

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-22 Thread Atri Sharma
> > However, I think the main issue isn't finding new algorithms that are > better in some specific circumstances. The hard part is figuring out > whether their performance is better in general. My idea was to create > a patch to capture page pinning traffic from PostgreSQL (maybe stream > out into

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-22 Thread Atri Sharma
On Fri, Mar 22, 2013 at 4:53 PM, Amit Kapila wrote: > On Friday, March 22, 2013 4:36 PM Atri Sharma wrote: >> > >> > What would you do if the only young page has usage count zero during >> second >> > sweep. >> >> UmmThe same approach we take w

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-22 Thread Atri Sharma
> > What would you do if the only young page has usage count zero during second > sweep. UmmThe same approach we take when there is no page with usage count zero in a sweep in the current algorithm? > I don't think introducing another factor along with usage count would do any > much help.

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-22 Thread Atri Sharma
> >> I think that if the initialization of USAGE_COUNT starts at the maximum >> allowed value instead of one, we can have a better solution to this >> problem. > > So what is your idea, if you start at maximum, what we will do for further > accesses to it? I havent chalked out a detailed plan yet,

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-21 Thread Atri Sharma
Sent from my iPad On 22-Mar-2013, at 11:28, Amit Kapila wrote: > On Friday, March 22, 2013 10:22 AM Atri Sharma wrote: >> Hello all, >> >> Sorry if this is a naive question. >> >> I was going through Greg Smith's slides on buffer >> cache(http:

[HACKERS] Page replacement algorithm in buffer cache

2013-03-21 Thread Atri Sharma
Hello all, Sorry if this is a naive question. I was going through Greg Smith's slides on buffer cache(http://www.westnet.com/~gsmith/content/postgresql/InsideBufferCache.pdf). When going through the page replacement algorithm that we use i.e. clocksweep algorithm, I felt a potential problem in ou

Re: [HACKERS] [pgsql-advocacy] Call for Google Summer of Code mentors, admins

2013-03-20 Thread Atri Sharma
It does sound nice,something like cron? We can use a scheduling algorithm, and can define a pool of tasks as well as a time constraint for the amount of time which can be used for running the tasks.Then, a scheduling algorithm can pick tasks from the pool based on priorities and the time durati

Re: [HACKERS] [pgsql-advocacy] Call for Google Summer of Code mentors, admins

2013-02-23 Thread Atri Sharma
> Take a look at MADLib. My suggestion would be extending the MADlib > functions; there's plenty of unimplemented ML algrothims which could be > added to it. I went through the MADLib library and came up with the following two ideas which I feel could be potential GSoC 2013 projects: 1) MADlib c

Re: [HACKERS] [pgsql-advocacy] Call for Google Summer of Code mentors, admins

2013-02-15 Thread Atri Sharma
I would love to mentor if anybody would be willing to take a project in it. Atri Sent from my iPad On 15-Feb-2013, at 23:04, Josh Berkus wrote: > On 02/15/2013 06:03 AM, Atri Sharma wrote: >> Can't we have something related to machine learning? I was thinking of >> e

Re: [HACKERS] [pgsql-advocacy] Call for Google Summer of Code mentors, admins

2013-02-15 Thread Atri Sharma
Can't we have something related to machine learning? I was thinking of extending a technique where we can fill in some missing values in a data set for the user if he wants us to using some standard ml algorithms. Atri Sent from my iPad On 15-Feb-2013, at 19:26, Heikki Linnakangas wrote: > O

Re: [HACKERS] Call for Google Summer of Code mentors, admins

2013-02-14 Thread Atri Sharma
I forgot to mark all. I could be a co mentor, helping in overall coordination and supporting all the students in getting familiar with the project and community etc. Atri Sent from my iPad On 15-Feb-2013, at 8:34, Stephen Frost wrote: > * Josh Berkus (j...@agliodbs.com) wrote: >> - Who wants

Re: [HACKERS] [pgsql-advocacy] Call for Google Summer of Code mentors, admins

2013-02-14 Thread Atri Sharma
I forgot to mark all. I could be a co mentor, helping in overall coordination and supporting all the students in getting familiar with the project and community etc. Atri Sent from my iPad On 14-Feb-2013, at 23:32, Josh Berkus wrote: > Folks, > > Once again, Google is holding Summer of Code

Re: [HACKERS] Fractal tree indexing

2013-02-13 Thread Atri Sharma
Sent from my iPad On 13-Feb-2013, at 22:21, Heikki Linnakangas wrote: > On 13.02.2013 18:43, Andrew Dunstan wrote: >> >> On 02/13/2013 11:20 AM, Tom Lane wrote: >>> Heikki Linnakangas writes: The basic fractal indexes is what I've read on some presentation slides on the 'net, so

Re: [HACKERS] Fractal tree indexing

2013-02-13 Thread Atri Sharma
Yeah,it is just a fancy name for something that has nothing to do with fractals.I guess everything suave these days is fractal! That said,the buffered concept itself looks really cool and should help us in large data sets.I am eager to get off the mark with it. Will we be building the index fr

Re: [HACKERS] Fractal tree indexing

2013-02-13 Thread Atri Sharma
Sent from my iPad . > > That makes no sense. I don't see any way to implement this in an opclass, and > it wouldn't make sense to re-implement this for every opclass anyway. > > The basic idea of a fractal tree index is to attach a buffer to every > non-leaf page. On insertion, instead of des

Re: [HACKERS] Fractal tree indexing

2013-02-13 Thread Atri Sharma
Sent from my iPad On 13-Feb-2013, at 20:30, Tom Lane wrote: > > First explain why you couldn't build it as an opclass for gist or > spgist ... > > That needs thinking about a bit.I was confused about the current indexes because they all build on BTrees.But, building an opclass

Re: [HACKERS] Fractal tree indexing

2013-02-13 Thread Atri Sharma
> > > If they are patented as Alexander says upthread, then surely the idea is dead > in the water. > > True, I think so too. But,the generic WAL seems an awesome idea and I would love to help. Atri -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to

Re: [HACKERS] Fractal tree indexing

2013-02-13 Thread Atri Sharma
Sent from my iPad On 13-Feb-2013, at 19:31, Simon Riggs wrote: . > > I think its a good idea, the best idea even, but we still have no idea > what the requirements are without a clear case for an external index. > It could easily turn out that we invent a plausible API that's not > actually of

Re: [HACKERS] Fractal tree indexing

2013-02-13 Thread Atri Sharma
Sent from my iPad On 13-Feb-2013, at 19:05, Heikki Linnakangas wrote: > On 13.02.2013 15:31, Alexander Korotkov wrote: >> On Wed, Feb 13, 2013 at 4:51 PM, Greg Stark wrote: >> >>> Heikki was talking about a generic WAL record type that would just >>> store a binary delta between the version

Re: [HACKERS] Fractal tree indexing

2013-02-13 Thread Atri Sharma
Sent from my iPad On 13-Feb-2013, at 18:21, Greg Stark wrote > > > Heikki was talking about a generic WAL record type that would just > store a binary delta between the version of the block when it was > locked and when it was unlocked. That would handle any extension > cleanly as far as data

Re: [HACKERS] Fractal tree indexing

2013-02-13 Thread Atri Sharma
Wed: > > I remember we have already discussed fractal trees privately. Short > conclusions are so: > 1) Fractal tree indexes are patented. It is distributed as commercial > extension to MySQL. So we can't include it into PostgreSQL core. > 2) Tokutek can't provide full-fledged fractal tree indexes

Re: [HACKERS] Fractal tree indexing

2013-02-13 Thread Atri Sharma
On Wed, Feb 13, 2013 at 3:08 PM, Heikki Linnakangas wrote: > On 13.02.2013 11:01, Atri Sharma wrote: >> >> Hi all, >> >> Just a curiosity I couldnt control. I was recently reading about >> Fractal tree indexing >> (http://www.tokutek.com/2012/12/fractal-tre

[HACKERS] Fractal tree indexing

2013-02-13 Thread Atri Sharma
Hi all, Just a curiosity I couldnt control. I was recently reading about Fractal tree indexing (http://www.tokutek.com/2012/12/fractal-tree-indexing-overview/) and how TokuDB engine for MySQL is really working nicely with big data. I was wondering, do we have support for fractal tree indexing? I

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2013-01-18 Thread Atri Sharma
On 18-Jan-2013, at 17:04, Craig Ringer wrote: > On 12/14/2012 09:57 PM, Amit Kapila wrote: >>> >>> I need to validate the vacuum results. It's possible that this is >>> solvable by tweaking xmin check inside vacuum. Assuming that's fixed, >>> the question stands: do the results justify the cha

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-12-13 Thread Atri Sharma
On Thu, Dec 13, 2012 at 6:36 PM, Hari Babu wrote: > On Thu, Dec 7, 2012 at 7:56 PM, Hari babu > wrote: > > >>On Thu, Dec 6, 2012 at 8:52 PM, Merlin Moncure > wrote: > > >>Thanks for that -- that's fairly comprehensive I'd say. I'm quite > > ** ** > > >>interested in that benchmarki

Re: [HACKERS] Do we need so many hint bits?

2012-11-19 Thread Atri Sharma
On Tue, Nov 20, 2012 at 12:08 AM, Jeff Davis wrote: > On Mon, 2012-11-19 at 23:50 +0530, Atri Sharma wrote: > > > Sorry If I am being a bit naive, but shouldnt a simple mutex work in > > the case when a process wants to change the VM bit in cache? > > > > Mut

Re: [HACKERS] Do we need so many hint bits?

2012-11-19 Thread Atri Sharma
> > It's quite common to load a lot of data, and then do some reads for a > while (setting hint bits and flushing them to disk), and then do a > VACUUM a while later, setting PD_ALL_VISIBLE and writing all of the > pages again. Also, if I remember correctly, Robert went to significant > effort when

Re: [HACKERS] Do we need so many hint bits?

2012-11-19 Thread Atri Sharma
On Sun, Nov 18, 2012 at 10:43 PM, Jeff Davis wrote: > On Sun, 2012-11-18 at 15:19 +0100, Andres Freund wrote: > > On Sunday, November 18, 2012 03:07:01 AM Jeff Davis wrote: > > > Process A (process that clears a VM bit for a data page): > > > 1. Acquires exclusive lock on data buffer > > > 2.

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-11-16 Thread Atri Sharma
On Fri, Nov 16, 2012 at 7:33 PM, Merlin Moncure wrote: > On Fri, Nov 16, 2012 at 4:32 AM, Amit Kapila > wrote: > > On Thursday, November 15, 2012 10:19 PM Merlin Moncure wrote: > >> On Thu, Nov 15, 2012 at 10:25 AM, Amit Kapila > >> wrote: > > > >> > >> Sure, although in scans we are using ring

<    1   2   3   4   >