On 03/28/2014 11:29 AM, Serge Kovaleff wrote:
Hi Iliia,
I would take a look into BSON http://bsonspec.org/
Cheers,
Serge Kovaleff
On Thu, Mar 27, 2014 at 8:23 PM, Illia Khudoshyn
<ikhudos...@mirantis.com <mailto:ikhudos...@mirantis.com>> wrote:
Hi, Openstackers,
I'm currently working on adding bulk data load functionality to
MagnetoDB. This functionality implies inserting huge amounts of
data (billions of rows, gigabytes of data). The data being
uploaded is a set of JSON's (for now). The question I'm interested
in is a way of data transportation. For now I do streaming HTTP
POST request from the client side with gevent.pywsgi on the server
side.
Could anybody suggest any (better?) approach for the
transportation, please?
What are best practices for that.
Thanks in advance.
--
Best regards,
Illia Khudoshyn,
Software Engineer, Mirantis, Inc.
38, Lenina ave. Kharkov, Ukraine
www.mirantis.com <http://www.mirantis.ru/>
www.mirantis.ru <http://www.mirantis.ru/>
Skype: gluke_work
ikhudos...@mirantis.com <mailto:ikhudos...@mirantis.com>
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Hi Iliia,
I guess if we a talking about cassandra batch loading the fastest way is
to generate sstables locally and load it into Cassandra via JMX or
sstableloader
http://www.datastax.com/dev/blog/bulk-loading
If you want to implement bulk load via magnetodb layer (not to cassandra
directly) you could try to use simple tcp socket and implement your
binary protocol (using bson for example). Http is text protocol so using
tcp socket can help you to avoid overhead of base64 encoding. In my
opinion, working with HTTP and BSON is doubtful solution
because you wil use 2 phase encoddung and decoding: 1) "object to bson",
2) "bson to base64", 3) "base64 to bson", 4) "bson to object" 1) "obect
to json" instead of 1) "object to json", 2) "json to object" in case of
HTTP + json
Http streaming as I know is asynchronous type of http. You can expect
performance growing thanks to skipping generation of http response on
server side and waiting on for that response on client side for each
chunk. But you still need to send almost the same amount of data. So if
network throughput is your bottleneck - it doesn't help. If server side
is your bottleneck - it doesn't help too.
Also pay your attention that in any case, now MagnetoDB Cassandra
Storage convert your data to CQL query which is also text. It would be
nice to implement MagnetoDB BatchWriteItem operation via Cassandra
sstable generation and loading via sstableloader, but unfortunately as I
know this functionality support implemented only for Java world
--
Best regards,
Dmitriy Ukhlov
Mirantis Inc.
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
