On Mon, Apr 1, 2013 at 11:33 PM, James Taylor <jtay...@salesforce.com>wrote:
> Maybe if we can keep nullability separate from the > serialization/deserialization, we can come up with a solution that works? I think implied null could work, but let's build out the matrix. I see two kinds of types: fixed- and variable-width. These types are used in two scenarios: on their own or as part of a compound type. A fixed-width type used standalone can enfer null from absence of a value. When used in a compound type, absence isn't enough to indicate null unless it's the last value in the sequence. To support a null field in the middle of the compound type, it is forced to explicitly mark the field as null. The only solution I can think of (without sacrificing the full value range, per my original question) is to write the full type width bytes, followed by an isNull byte. Thus, for example, the INT type consumes 4 bytes when serialized stand-alone, but 5 bytes when composed. James, how does Phoenix handle a null fixed-width rowkey component? I don't see that implemented in PDataType enum. Variable-width used standalone are simple enough because HBase handles arbitrary length byte[]'s everywhere. Variable-width in composite is a problem. Phoenix forces these value to only appear as the last position in the composite, as I understand it. Orderly provides explicit null and termination bytes by taking advantage of a feature of UTF-8 encoding. Support for bytes is equally ugly (but clever) in that byte digits are encoded in BCD. Both of these approaches bloat slightly the serialized representation over the natural representation, but they allow the variable-length types to be used anywhere within the compound type. As an added bonus regarding code maintainability, their serialization entirely self-contained within the type. That's in contrast to the fixed-width type implementation described above, where null is explicitly encoded by the compound type. My opinion is the computational and storage overhead imposed by Orderly's implementation are worth the trade-off in flexibility in user consumption. Correct me if i'm wrong James, but you're saying, from your experience with Phoenix, users are willing to work within that constraint? Thanks, Nick On 04/01/2013 11:29 PM, Jesse Yates wrote: Actually, that isn't all that far-fetched of a format Matt - pretty common >> anytime anyone wants to do sortable lat/long (*cough* three letter >> agencies >> cough*). >> >> Wouldn't we get the same by providing a simple set of libraries (ala >> orderly + other HBase useful things) and then still give access to the >> underlying byte array? Perhaps a nullable key type in that lib makes sense >> if lots of people need it and it would be nice to have standard libraries >> so tools could interop much more easily. >> ------------------- >> Jesse Yates >> @jesse_yates >> jyates.github.com >> >> >> On Mon, Apr 1, 2013 at 11:17 PM, Matt Corgan <mcor...@hotpads.com> wrote: >> >> Ah, I didn't even realize sql allowed null key parts. Maybe a goal of >>> the >>> interfaces should be to provide first-class support for custom user types >>> in addition to the standard ones included. Part of the power of hbase's >>> plain byte[] keys is that users can concoct the perfect key for their >>> data >>> type. For example, I have a lot of geographic data where I interleave >>> latitude/longitude bits into a sortable 64 bit value that would probably >>> never be included in a standard library. >>> >>> >>> On Mon, Apr 1, 2013 at 8:38 PM, Enis Söztutar <enis....@gmail.com> >>> wrote: >>> >>> I think having Int32, and NullableInt32 would support minimum overhead, >>>> >>> as >>> >>>> well as allowing SQL semantics. >>>> >>>> >>>> On Mon, Apr 1, 2013 at 7:26 PM, Nick Dimiduk <ndimi...@gmail.com> >>>> wrote: >>>> >>>> Furthermore, is is more important to support null values than squeeze >>>>> >>>> all >>> >>>> representations into minimum size (4-bytes for int32, &c.)? >>>>> On Apr 1, 2013 4:41 PM, "Nick Dimiduk" <ndimi...@gmail.com> wrote: >>>>> >>>>> On Mon, Apr 1, 2013 at 4:31 PM, James Taylor <jtay...@salesforce.com >>>>>> wrote: >>>>>> >>>>>> From the SQL perspective, handling null is important. >>>>>>> >>>>>> >>>>>> From your perspective, it is critical to support NULLs, even at the >>>>>> expense of fixed-width encodings at all or supporting representation >>>>>> >>>>> of a >>>> >>>>> full range of values. That is, you'd rather be able to represent NULL >>>>>> >>>>> than >>>>> >>>>>> -2^31? >>>>>> >>>>>> On 04/01/2013 01:32 PM, Nick Dimiduk wrote: >>>>>> >>>>>>> Thanks for the thoughtful response (and code!). >>>>>>>> >>>>>>>> I'm thinking I will press forward with a base implementation that >>>>>>>> >>>>>>> does >>>> >>>>> not >>>>>>>> support nulls. The idea is to provide an extensible set of >>>>>>>> >>>>>>> interfaces, >>>> >>>>> so I >>>>>>>> think this will not box us into a corner later. That is, a >>>>>>>> >>>>>>> mirroring >>> >>>> package could be implemented that supports null values and accepts >>>>>>>> the relevant trade-offs. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Nick >>>>>>>> >>>>>>>> On Mon, Apr 1, 2013 at 12:26 PM, Matt Corgan <mcor...@hotpads.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>> I spent some time this weekend extracting bits of our >>>>>>>> >>>>>>> serialization >>> >>>> code to >>>>>>>>> a public github repo at >>>>>>>>> http://github.com/hotpads/****data-tools<http://github.com/hotpads/**data-tools> >>>>>>>>> < >>>>>>>>> >>>>>>>> http://github.com/hotpads/**data-tools<http://github.com/hotpads/data-tools> >>>>> > >>>>> >>>>>> . >>>>>>>>> Contributions are welcome - i'm sure we all have this stuff >>>>>>>>> >>>>>>>> laying >>> >>>> around. >>>>>>>>> >>>>>>>>> You can see I've bumped into the NULL problem in a few places: >>>>>>>>> * >>>>>>>>> >>>>>>>>> https://github.com/hotpads/****data-tools/blob/master/src/**<https://github.com/hotpads/**data-tools/blob/master/src/**> >>>>>>>>> main/java/com/hotpads/data/****primitive/lists/LongArrayList.** >>>>>>>>> **java< >>>>>>>>> >>>>>>>> https://github.com/hotpads/**data-tools/blob/master/src/** >>> main/java/com/hotpads/data/**primitive/lists/LongArrayList.**java<https://github.com/hotpads/data-tools/blob/master/src/main/java/com/hotpads/data/primitive/lists/LongArrayList.java> >>> >>>> * >>>>>>>>> >>>>>>>>> https://github.com/hotpads/****data-tools/blob/master/src/**<https://github.com/hotpads/**data-tools/blob/master/src/**> >>>>>>>>> main/java/com/hotpads/data/****types/floats/DoubleByteTool.**** >>>>>>>>> java< >>>>>>>>> >>>>>>>> https://github.com/hotpads/**data-tools/blob/master/src/** >>> main/java/com/hotpads/data/**types/floats/DoubleByteTool.**java<https://github.com/hotpads/data-tools/blob/master/src/main/java/com/hotpads/data/types/floats/DoubleByteTool.java> >>> >>>> Looking back, I think my latest opinion on the topic is to reject >>>>>>>>> nullability as the rule since it can cause unexpected behavior and >>>>>>>>> confusion. It's cleaner to provide a wrapper class (so both >>>>>>>>> LongArrayList >>>>>>>>> plus NullableLongArrayList) that explicitly defines the behavior, >>>>>>>>> >>>>>>>> and >>>> >>>>> costs >>>>>>>>> a little more in performance. If the user can't find a pre-made >>>>>>>>> >>>>>>>> wrapper >>>>> >>>>>> class, it's not very difficult for each user to provide their own >>>>>>>>> interpretation of null and check for it themselves. >>>>>>>>> >>>>>>>>> If you reject nullability, the question becomes what to do in >>>>>>>>> >>>>>>>> situations >>>>> >>>>>> where you're implementing existing interfaces that accept nullable >>>>>>>>> params. >>>>>>>>> The LongArrayList above implements List<Long> which requires an >>>>>>>>> add(Long) >>>>>>>>> method. In the above implementation I chose to swap nulls with >>>>>>>>> Long.MIN_VALUE, however I'm now thinking it best to force the user >>>>>>>>> >>>>>>>> to >>>> >>>>> make >>>>>>>>> that swap and then throw IllegalArgumentException if they pass >>>>>>>>> >>>>>>>> null. >>> >>>> >>>>>>>>> On Mon, Apr 1, 2013 at 11:41 AM, Doug Meil < >>>>>>>>> doug.m...@explorysmedical.com >>>>>>>>> >>>>>>>>> wrote: >>>>>>>>>> HmmmŠ good question. >>>>>>>>>> >>>>>>>>>> I think that fixed width support is important for a great many >>>>>>>>>> >>>>>>>>> rowkey >>>> >>>>> constructs cases, so I'd rather see something like losing >>>>>>>>>> >>>>>>>>> MIN_VALUE >>> >>>> and >>>>> >>>>>> keeping fixed width. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 4/1/13 2:00 PM, "Nick Dimiduk" <ndimi...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>> Heya, >>>>>>>>>> >>>>>>>>>>> Thinking about data types and serialization. I think null >>>>>>>>>>> >>>>>>>>>> support >>> >>>> is >>>> >>>>> an >>>>>>>>>>> important characteristic for the serialized representations, >>>>>>>>>>> especially >>>>>>>>>>> when considering the compound type. However, doing so in >>>>>>>>>>> >>>>>>>>>> directly >>> >>>> incompatible with fixed-width representations for numerics. For >>>>>>>>>>> >>>>>>>>>>> instance, >>>>>>>>>> if we want to have a fixed-width signed long stored on 8-bytes, >>>>>>>>>> >>>>>>>>> where >>>> >>>>> do >>>>>>>>>>> you put null? float and double types can cheat a little by >>>>>>>>>>> >>>>>>>>>> folding >>> >>>> negative >>>>>>>>>>> and positive NaN's into a single representation (this isn't >>>>>>>>>>> >>>>>>>>>> strictly >>>> >>>>> correct!), leaving a place to represent null. In the long >>>>>>>>>>> >>>>>>>>>> example >>> >>>> case, >>>>>>>>>>> the >>>>>>>>>>> obvious choice is to reduce MAX_VALUE or increase MIN_VALUE by >>>>>>>>>>> >>>>>>>>>> one. >>>> >>>>> This >>>>>>>>>>> will allocate an additional encoding which can be used for null. >>>>>>>>>>> >>>>>>>>>> My >>>> >>>>> experience working with scientific data, however, makes me wince >>>>>>>>>>> >>>>>>>>>> at >>>> >>>>> the >>>>>>>>>>> idea. >>>>>>>>>>> >>>>>>>>>>> The variable-width encodings have it a little easier. There's >>>>>>>>>>> >>>>>>>>>> already >>>>> >>>>>> enough going on that it's simpler to make room. >>>>>>>>>>> >>>>>>>>>>> Remember, the final goal is to support order-preserving >>>>>>>>>>> >>>>>>>>>> serialization. >>>>> >>>>>> This >>>>>>>>>>> imposes some limitations on our encoding strategies. For >>>>>>>>>>> >>>>>>>>>> instance, >>> >>>> it's >>>>>>>>>>> not >>>>>>>>>>> enough to simply encode null, it really needs to be encoded as >>>>>>>>>>> >>>>>>>>>> 0x00 >>>> >>>>> so >>>>> >>>>>> as >>>>>>>>>> to sort lexicographically earlier than any other value. >>>>>>>>>> >>>>>>>>>>> What do you think? Any ideas, experiences, etc? >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Nick >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >
