Hey, Travis. Apologies for missing your reply on Friday. (Lost in the flood, I'm afraid.) Thank you for reviewing the proposal.
> How large are the partitions that they're causing issues for you? I hadn't measured. (Thank you for doing so.) But given the large volume of partition-related events we're slated to consume shortly in Oozie, we figured it'd be in our best interest to shed any information that can be queried from the metastore. The parameter-set, location-related details etc. seemed like too much to be carrying on each event. (We're still reviewing with Oozie that the proposed info works for them.) > Seems like messages would still deserialize on the other side if fields are >added, clients just wouldn't know about the new field? Another good question. I'd initially figured that while the message might be received, the Partition object would not be deserialized correctly. In hindsight, that might not be. Here're some thoughts though: 1. We'd like to decouple the notification-format from the layout of the Partition class, with the notification containing just enough to identify the partition. This way, changes to the Partition-class don't change the notifications at all. 2. It's conceivable that we might add further information to a notification, beyond what might be contained in a Partition object. 3. We could have chosen a Thrift definition for the notification. JSON seemed simpler. My follow-up to this change would be to add support for adding a group of Partitions as an atomic "set". The AddPartitionEvent could conceivably contain a collection of partition-specs, all corresponding to the same set. (The current handling of the use-case is hacky and broken.) Adding a large number of Partition-thrift-objects to the same notification would be wasteful. The patch up on JIRA attempts to support future serialization schemes, while providing the JSON implementation. Does that sound reasonable? Mithun ________________________________ From: Travis Crawford <[email protected]> To: [email protected]; Mithun Radhakrishnan <[email protected]> Sent: Friday, November 2, 2012 8:23 AM Subject: Re: Add/delete-partition JMS message format proposal. Hey Mithun - Cool that you're giving the oozie integration a bit more love. A few questions inline: On Tue, Oct 30, 2012 at 2:26 PM, Mithun Radhakrishnan <[email protected]> wrote: Hello, HCat-Dev. > >I'm working on modifying the HCat messages (sent over JMS/ActiveMQ, for >partition-add/delete) so that clients (such as >Oozie) would have an easier time with consumption. >Here are some limitations of what's available currently: >1. The present implementation in HCatalog (branch-0.4/) seems to send the >entire Partition (Java) instance in serialized fashion. Since the >partition-parameters, hdfs-location etc. are all serialized, the messages are >rather, emm, garrulous. > Correct, the messages sent over jms include an instance of org.apache.hadoop.hive.metastore.api.Partition. Eventually oozie needs this info. It sounds like this object is larger than you want to publish over jms, and would rather fetch it yourself on the other side? How large are the partitions that they're causing issues for you? I did a quick test in NotificationListener.onAddPartition and see the following when running the test: ByteArrayOutputStream baos = new ByteArrayOutputStream(); ObjectOutputStream oos = new ObjectOutputStream(baos); oos.writeObject(partition); oos.close(); LOG.info("Size of partition: " + baos.size()); [main] INFO org.apache.hcatalog.listener.NotificationListener - Size of partition: 794 794 bytes doesn't seem large enough to introduce additional complexity over. Are you seeing drastically different object sizes? 2. There doesn't seem to be any support for versioning either. So when new fields are added, older clients won't work at all without update. > Are you referring to the message map that's actually sent over the wire? I'm not super familiar with jms, so sorry if this is a dumb question. Seems like messages would still deserialize on the other side if fields are added, clients just wouldn't know about the new field? If this is the case, how does switching to something like json help? I'm initially hesitant to introduce yet another serialization format to keep track of, because of the added complexity & maintenance issues. Definitely not opposed if its the right thing to do, but am curious to learn a bit more about this before diving in. --travis >Could we consider transmitting only that info which identifies the partitions >that pertain to the operation (e.g. partition keys), and drop any information >that might be gathered from querying the metadata (e.g. storage location, >partition-parameters, etc.) > >We're also considering that the initial implementation encode the ActiveMQ >payload in JSON. Here's an example of the proposed message format for an >"add_partition" operation: > >"add_partition": { > "hcat_server" : "thrift://my.hcat.server:9080", > "hcat_service_principal" : "hcat/[email protected]", > "db": "default", > "table": "starling_jobs", > "partitions": > [ > {"grid": "AxoniteBlue", "dt": "2012_10_25"},// Sets of partition-keys. > {"grid": "AxoniteBlue", "dt": "2012_10_26"}, > {"grid": "AxoniteBlue", "dt": "2012_10_27"}, > {"grid": "AxoniteBlue", "dt": "2012_10_28"}, > ], > "timestamp": "1351534729" // In this case, interpreted as creation-time. >} > >If we continue to use JMS MapMessages, we could consider having 3 keys in the >map: >1. version = "1" (for the first implementation. Increment as we go.) >2. format = "json" (We could consider adding different formats if we choose.) >3. message = <the json message body, as above.> > >The version and format help a factory choose the right implementation to >deserialize the message. (A client-side library we supply to Oozie should hide >this and provide POJOs.) > >Since the "partitions" field is an array, and since the values corresponding >to partition-keys are all strings, we'd be able to accommodate partial >partitions-specs, or even wild-cards. This might help us add support for >"mark-set-done" later on. > >The first key ("add_partition", "drop_partition" or "alter_partition") >indicates the operation, and the value indicates the record-body. (At first >glance, the record-body doesn't change for these operations. But that might >change, so we'll keep them distinct.) > >Also note that HiveMetaStore::add_partitions_core() currently doesn't send 1 >message for the entire set of partitions being added. Instead we get one >message per partition. This could be verbose and sub-optimal. We'll tackle >this sort of thing after we've nailed the format down. > >I'm toying with the idea of adding an "other" property, an array of key-values >to accommodate stuff we hadn't considered, at "run-time" (like if we want to >introduce a hack). The need for such a property is contingent on the behaviour >of Jackson w.r.t. newly added properties in the record-body. (I'll run >experiments and keep you posted.) > >What do you think? > >Mithun > >
