I asked because the IBM guys have some code to be released soon that will
address some of these issues.

In particular, Kevin Breyer was at the HUG and we talked a bit about their
format.  Some important characteristics are:

- it is pretty much a binary JSON implementation

- value parsing doesn't happen because it is a binary format.  That means
that integers, doubles and dates can be read much more quickly.

- names are repeated in the data just as in JSON, but are easily compressed
away with any compression technique you care to use.

- he says it is fast, but I haven't tried it yet.

So this will address the first point very directly, will indirectly address
the second point, and the third point may be FUD.

Relative to thrift, JSON has the advantage of not requiring a schema as well
as the disadvantage of not having a schema.  The advantage is that the data
is more fluid and I don't have to generate code to handle the records.  The
disadvantage is that I lose some data completeness and typing guarantees.
On balance, I would like to use JSON-like data quite a bit in ad hoc data
streams and in logs where the producer and consumer of the data are not
visible to parts of the data processing chain.

I am very excited to see the IBM code.

On 5/23/08 6:37 AM, "Stuart Sierra" <[EMAIL PROTECTED]> wrote:

> On 5/22/08 1:54 PM, "Stuart Sierra" <[EMAIL PROTECTED]> wrote:
>> I've tried using JSON to store structured data in TextOutputFormat,
>> which works but is not very efficient.  Any better suggestions?
> 
> On Thu, May 22, 2008 at 5:21 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:
>> What is it that makes you not think JSON has to be inefficient?
>> ? Repeated value parsing ?
>> ? Redundant redundant data labels ?
>> ? Generic "parsing must be slow" prejudice ?
> 
> All three :) but it's not a strong prejudice.  I was just wondering if
> there's a better way.  Is it worthwhile, for example, to write my own
> record class that implements Writable?
> 
> -Stuart

Reply via email to