Just intuition here, but embedding postrges inside Assimilation seems heavy for just JSON. Of course, you already embed neo4j so you know how to handle having a dependency on a large subsystem.
Then, there's mongodb vs postgres vs ...: https://www.sisense.com/blog/postgres-vs-mongodb-for-storing-json-data/ https://www.quora.com/Which-is-better-storing-json-objects-in-json-files-in-Redis-or-MongoDB-RethinkDB On 9/2/18, 10:05, "Assimilation on behalf of Alan Robertson" <assimilation-boun...@lists.community.tummy.com on behalf of al...@unix.sh> wrote: EXTERNAL EMAIL – Use caution with any links or file attachments. This data is generated by a variety of shell scripts that do discovery - potentially dozens of them - and each is different. Some of the most critical data is decomposed to attributes - but not most of it. -- Alan Robertson al...@unix.sh On Fri, Feb 9, 2018, at 2:58 AM, Michael Hunger wrote: > I think this is ok. > I wished we had full document support yet. > > I know that pg has really good jsonb support, so go for it. > > Did you ever try to destrucure the data into properties? Not sure how > deeply nested it is? And leave off all that are just defaults > > Von meinem iPhone gesendet > > > Am 09.02.2018 um 04:10 schrieb Alan Robertson <al...@unix.sh>: > > > > Hi, > > > > There is one set of data that when I insert it into Neo4j - it's really, really slow. It's discovery data - which is JSON - and sometimes very large - a few megabytes. Many of them are smallish, but having items a few kilobytes is common, and dozens of kilobytes is also common, and some few things are in the megabyte+ range. [Because of compression, I can send up to 3 megabytes of this JSON over UDP]. > > > > There are a few things I can do with Neo4j to make inserting it faster, but I don't think a lot -- and when I get done, the data is very hard to query against (it involves regexes against unindexed data, and is a performance nightmare). > > > > Postgres has JSON support, and it has real transactions and a reputation for being very solid. I did some benchmarking and it is a a couple of orders of magnitude faster than Neo4j with both of them untuned. In addition, Postgres JSON (jsonb) can have indexes over the JSON information - greatly improving the query capabilities over what Neo4j can do for this same data. > > > > I'm not thinking about doing anything except moving this one class of data to Postgres. This particular class of data is also idempotent, which has advantages when you have multiple databases involved... > > > > Since this particular type of data is its own object in the Python, having it be in Postgres wouldn't likely be horrible to implement. > > > > If I'm going to do this in the next year or two, it makes sense to couple it with the rest of the backwards-incompatible changes I'm already putting into release 2. > > > > Does anyone think this is a show-stopper to use two databases? > > > > -- > > Alan Robertson > > al...@unix.sh _______________________________________________ Assimilation mailing list - Discovery-Driven Monitoring Assimilation@lists.community.tummy.com http://lists.community.tummy.com/cgi-bin/mailman/listinfo/assimilation http://assimmon.org/ Notice: This e-mail message, together with any attachments, contains information of Merck & Co., Inc. (2000 Galloping Hill Road, Kenilworth, New Jersey, USA 07033), and/or its affiliates Direct contact information for affiliates is available at http://www.merck.com/contact/contacts.html) that may be confidential, proprietary copyrighted and/or legally privileged. It is intended solely for the use of the individual or entity named on this message. If you are not the intended recipient, and have received this message in error, please notify us immediately by reply e-mail and then delete it from your system. _______________________________________________ Assimilation mailing list - Discovery-Driven Monitoring Assimilation@lists.community.tummy.com http://lists.community.tummy.com/cgi-bin/mailman/listinfo/assimilation http://assimmon.org/
