Hi,
During our work on a distributed processing system (which uses SQLite
shards), we have hit the SQLITE_MAX_ATTACHED limit of attached DBs.
The way we use SQLite for distributed processing [*], is the following:
- Each table is sharded into multiple SQLite DBs on different nodes of
the cluster.
- To process a query, we run on each shard a query which produces
multiple sharded SQLite result DBs.
- We redistribute in the cluster the result DBs, and the next set of
cluster nodes, attaches all the input shard SQLite DBs, and it creates a
temp view that unions all the input DB shards into a single view.
- It then executes a query on the views that produces new result DB shards
- and so on
We recently got access to a cluster of 64 nodes and it is very easy now
to hit the SQLITE_MAX_ATTACHED limit (1 DB shard gets produced per node).
So the question that i have is:
Is there any way to go beyond the SQLITE_MAX_ATTACHED limit for *read
only* attached DBs?
Also is there anyway for SQLite to create an automatic index on a view
(or Virtual Table), without having to first materialize the view (or VT)?
Thanks in advance.
Lefteris Stamatogiannakis.
[*] The same processing ideas are used in hadapt:
http://hadapt.com/
which uses Postgres for the DB shards.
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
