For the XML side of things I was just thinking that if someone wanted to export a large store (such as 1GB+, etc) then it would produce very large XML files which would not be efficient to store or transfer. A use case I had in mind is exporting a broker into XML on one machine and then transferring the data to another machine with an Artemis broker would be slow with large XML.
Gzip certainly works. I wasn't thinking of doing anything crazy with compression. What I was thinking of was just doing something simple like have an optional flag the user could set for the command that would just end up creating a GZIPOutputStream and write the XML directly to a gzip file which would save time (and disk space) from having to create the XML first and then compress it after the fact. On Thu, Feb 2, 2017 at 9:38 AM, Clebert Suconic <[email protected]> wrote: > On Thu, Feb 2, 2017 at 9:28 AM, Justin Bertram <[email protected]> > wrote: > >> For large stores I think we will need to do something here in terms of > compression otherwise we could end up with huge XML files. > > > > What specifically are your concerns and ideas here? Storage appears > cheap and plentiful and files can be compressed after they are created with > gzip or similar. Also, huge message stores are (IMO) an anti-pattern. A > message broker is not a database after all. What use-cases do you have in > mind? > > > I am not sure what CShanon had in mind, > > but we could (I"m not saying yet we should.. just listing the possibility): > > - import from KahaDB and do directly what the importer is doing. > Instead of dumping an XML, it could add directly to the broker running > from the kahaDB storage. > > > I don't mind a huge XML (or compressed gz in such way the importer > would read compressed). But it could be an option eventually. > > @CShanon: I wouldn't do the directly unless you are aware of user > demand, meaning the XML is not good for certain cases. >
