[
https://issues.apache.org/jira/browse/COUCHDB-1092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007948#comment-13007948
]
Filipe Manana commented on COUCHDB-1092:
----------------------------------------
Paul,
I'm not in any way sacrificing correctness, or at least intentionally.
1) When we received a document for writing:
I JSON decode it, take out all metadata (these 2 steps are done in current
trunk), and then JSON encode the remaining (user data) - this is what gets
written to disk with an append_term_md5 couch_file call.
2) When a document read request comes (or we need to send it to the view
server, via replicator, etc):
step 2.1) We read that JSON body binary (user data) we previously wrote - if
the md5 check fails, it's corrupted, and the process stops here. If the md5
check succeeds, I assemble an EJSON object with the metadata only (_id, _rev,
_conflicts, _deleted, etc), JSON encode it, and get a binary - since it's an
EJSON object it must get encoded as a JSON object, that is, the resulting
binary ends with a "}", and I take out that "}" getting "Prefix"
step 2.2) I grab the JSON body binary, and since it was the result of an EJSON
object decoding (in the write phase), I assume its very first byte is a "{",
take it out and get "Body".
step 2.3) I do Meta ++ Body
Where here do we room for some sort of SQL/JSON injection? Can you exemplify?
Is it a result of the JSON_ENCODE(MetaEJson) or the JSON_ENCODE(UserData)?
And yes, I think your jsonsplice operation is doing about half of the work of
an encoder.
If you're unhappy with json_stream_parse.erl, that is another separate
discussion in my opinion we can discuss that via mailing list or another ticket.
> Storing documents bodies as raw JSON binaries instead of serialized JSON terms
> ------------------------------------------------------------------------------
>
> Key: COUCHDB-1092
> URL: https://issues.apache.org/jira/browse/COUCHDB-1092
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Reporter: Filipe Manana
> Assignee: Filipe Manana
>
> Currently we store documents as Erlang serialized (via the term_to_binary/1
> BIF) EJSON.
> The proposed patch changes the database file format so that instead of
> storing serialized
> EJSON document bodies, it stores raw JSON binaries.
> The github branch is at:
> https://github.com/fdmanana/couchdb/tree/raw_json_docs
> Advantages:
> * what we write to disk is much smaller - a raw JSON binary can easily get up
> to 50% smaller
> (at least according to the tests I did)
> * when serving documents to a client we no longer need to JSON encode the
> document body
> read from the disk - this applies to individual document requests, view
> queries with
> ?include_docs=true, pull and push replications, and possibly other use
> cases.
> We just grab its body and prepend the _id, _rev and all the necessary
> metadata fields
> (this is via simple Erlang binary operations)
> * we avoid the EJSON term copying between request handlers and the db updater
> processes,
> between the work queues and the view updater process, between replicator
> processes, etc
> * before sending a document to the JavaScript view server, we no longer need
> to convert it
> from EJSON to JSON
> The changes done to the document write workflow are minimalist - after JSON
> decoding the
> document's JSON into EJSON and removing the metadata top level fields (_id,
> _rev, etc), it
> JSON encodes the resulting EJSON body into a binary - this consumes CPU of
> course but it
> brings 2 advantages:
> 1) we avoid the EJSON copy between the request process and the database
> updater process -
> for any realistic document size (4kb or more) this can be very expensive,
> specially
> when there are many nested structures (lists inside objects inside lists,
> etc)
> 2) before writing anything to the file, we do a term_to_binary([Len, Md5,
> TheThingToWrite])
> and then write the result to the file. A term_to_binary call with a binary
> as the input
> is very fast compared to a term_to_binary call with EJSON as input (or
> some other nested
> structure)
> I think both compensate the JSON encoding after the separation of meta data
> fields and non-meta data fields.
> The following relaximation graph, for documents with sizes of 4Kb, shows a
> significant
> performance increase both for writes and reads - especially reads.
> http://graphs.mikeal.couchone.com/#/graph/698bf36b6c64dbd19aa2bef63400b94f
> I've also made a few tests to see how much the improvement is when querying a
> view, for the
> first time, without ?stale=ok. The size difference of the databases (after
> compaction) is
> also very significant - this change can reduce the size at least 50% in
> common cases.
> The test databases were created in an instance built from that experimental
> branch.
> Then they were replicated into a CouchDB instance built from the current
> trunk.
> At the end both databases were compacted (to fairly compare their final
> sizes).
> The databases contain the following view:
> {
> "_id": "_design/test",
> "language": "javascript",
> "views": {
> "simple": {
> "map": "function(doc) { emit(doc.float1, doc.strings[1]); }"
> }
> }
> }
> ## Database with 500 000 docs of 2.5Kb each
> Document template is at:
> https://github.com/fdmanana/couchdb/blob/raw_json_docs/doc_2_5k.json
> Sizes (branch vs trunk):
> $ du -m couchdb/tmp/lib/disk_json_test.couch
> 1996 couchdb/tmp/lib/disk_json_test.couch
> $ du -m couchdb-trunk/tmp/lib/disk_ejson_test.couch
> 2693 couchdb-trunk/tmp/lib/disk_ejson_test.couch
> Time, from a user's perpective, to build the view index from scratch:
> $ time curl
> http://localhost:5984/disk_json_test/_design/test/_view/simple?limit=1
> {"total_rows":500000,"offset":0,"rows":[
> {"id":"0000076a-c1ae-4999-b508-c03f4d0620c5","key":null,"value":"wfxuF3N8XEK6"}
> ]}
> real 6m6.740s
> user 0m0.016s
> sys 0m0.008s
> $ time curl
> http://localhost:5985/disk_ejson_test/_design/test/_view/simple?limit=1
> {"total_rows":500000,"offset":0,"rows":[
> {"id":"0000076a-c1ae-4999-b508-c03f4d0620c5","key":null,"value":"wfxuF3N8XEK6"}
> ]}
> real 15m41.439s
> user 0m0.012s
> sys 0m0.012s
> ## Database with 100 000 docs of 11Kb each
> Document template is at:
> https://github.com/fdmanana/couchdb/blob/raw_json_docs/doc_11k.json
> Sizes (branch vs trunk):
> $ du -m couchdb/tmp/lib/disk_json_test_11kb.couch
> 1185 couchdb/tmp/lib/disk_json_test_11kb.couch
> $ du -m couchdb-trunk/tmp/lib/disk_ejson_test_11kb.couch
> 2202 couchdb-trunk/tmp/lib/disk_ejson_test_11kb.couch
> Time, from a user's perpective, to build the view index from scratch:
> $ time curl
> http://localhost:5984/disk_json_test_11kb/_design/test/_view/simple?limit=1
> {"total_rows":100000,"offset":0,"rows":[
> {"id":"00001511-831c-41ff-9753-02861bff73b3","key":null,"value":"2fQUbzRUax4A"}
> ]}
> real 4m19.306s
> user 0m0.008s
> sys 0m0.004s
> $ time curl
> http://localhost:5985/disk_ejson_test_11kb/_design/test/_view/simple?limit=1
> {"total_rows":100000,"offset":0,"rows":[
> {"id":"00001511-831c-41ff-9753-02861bff73b3","key":null,"value":"2fQUbzRUax4A"}
> ]}
> real 18m46.051s
> user 0m0.008s
> sys 0m0.016s
> All in all, I haven't seen yet any disadvantage with this approach. Also, the
> code changes
> don't bring additional complexity. I say the performance and disk space gains
> it gives are
> very positive.
> This branch still needs to be polished in a few places. But I think it isn't
> far from getting mature.
> Other experiments that can be done are to store view values as raw JSON
> binaries as well (instead of EJSON)
> and optional compression of the stored JSON binaries (since it's pure text,
> the compression ratio is very high).
> However, I would prefer to do these other 2 suggestions in separate
> branches/patches - I haven't actually tested
> any of them yet, so maybe they not bring significant gains.
> Thoughts? :)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
