Re: Very, very large .dat files

[Peter Hansson]

Thank for taking time to answer. Appreciated.

You basically confirmed my understanding: I need to go for a DIU 
approach if I want table partitioning.


On 05-Dec-17 17:30, Bryan Pendleton wrote:

You are correct, Derby does not provide table partitioning features
such as those provided by Oracle.

And you are correct, a single Derby table is a single .dat file.

But modern filesystems handle very large files without problems. Is
there a particular reason that you think a very large file will be a
problem for you? Or that you think a larger number of smaller .dat
files would have some benefit?

I believe there are a variety of library/framework approaches to
splitting up a logical table into multiple physical tables. People
often do this with data that is oriented around "time", and becomes of
less interest as it ages. I've seen applications which, e.g., create a
new table each day, named appropriately, and "rotate" these tables
over time so that the old data ages out and is dropped (by dropping
the older no-longer-wanted tables).

Careful use of CREATE VIEW can shield your application from most of
the impact of such techniques.

thanks,

bryan

Re: Very, very large .dat files

[Bryan Pendleton]

You are correct, Derby does not provide table partitioning features
such as those provided by Oracle.

And you are correct, a single Derby table is a single .dat file.

But modern filesystems handle very large files without problems. Is
there a particular reason that you think a very large file will be a
problem for you? Or that you think a larger number of smaller .dat
files would have some benefit?

I believe there are a variety of library/framework approaches to
splitting up a logical table into multiple physical tables. People
often do this with data that is oriented around "time", and becomes of
less interest as it ages. I've seen applications which, e.g., create a
new table each day, named appropriately, and "rotate" these tables
over time so that the old data ages out and is dropped (by dropping
the older no-longer-wanted tables).

Careful use of CREATE VIEW can shield your application from most of
the impact of such techniques.

thanks,

bryan

Re: Very, very large .dat files

[Peter Hansson]

I guess the question is if Apache Derby supports table partitioning (the 
term used in Oracle RDBMS).  I understand it doesn't.


This means that if I have an incredible large Derby table then that will 
mean an incredible large single .dat file too. (a table is always stored 
in a single .dat file, right?)


Any suggestions? Any props that may control this?


On 05-Dec-17 13:28, Peter Hansson wrote:


We've seen .dat files  in seg0 directory grow to several hundreds of 
gigabytes. While everything works ok such a file becomes unmanageable 
from an OS point of view. Is there a way to control when Derby starts 
a new conglomerate?  ... so that there are more .dat files but each of 
them of less size ?


Thanks.

Very, very large .dat files

[Peter Hansson]

We've seen .dat files  in seg0 directory grow to several hundreds of 
gigabytes. While everything works ok such a file becomes unmanageable 
from an OS point of view. Is there a way to control when Derby starts a 
new conglomerate?  ... so that there are more .dat files but each of 
them of less size ?


Thanks.