Subject: [PERFORM] What is best way to stream terabytes of
data into postgresql?
Preferably via JDBC, but by C/C++ if necessary.
Streaming being the operative word.
Tips appreciated.
Hi,
We granted our Java Loader to the Bizgres Open Source,
http://www.bizgres.org/assets/BZ_userguide.htm#50413574_pgfId-110126
You can load from STDIN instead of a file, as long as you prepend the
stream with the Loader Control file, for example:
for name in customer orders lineitem partsupp supplier part;do;cat
TPCH_load_100gb_${name}.ctl /mnt/remote-host/TPCH-Data/${name}.tbl.* |
loader.sh -h localhost -p 10001 -d tpch -t -u mpp; done
You can also run the loader from a remote host as well, with the -h
host being the target system with the Postgres database.
If you have terabytes of data, you might want to set a batch size (-b
switch) to commit occasionally.
Feel free to contact me directly if you have questions.
Thanks,
Frank
Frank Wosczyna
Systems Engineer
Greenplum / Bizgres MPP
www.greenplum.com
---(end of broadcast)---
TIP 5: don't forget to increase your free space map settings
