Quick update on this issue. I think I finally figured out a way to bypass Avro style "handshake" when employing "custom" format or content type implementation but still trying to reuse or benefit from the Serialize/Deserialize capabilities of Avro. Perhaps the following usage is the "intended" REUSE use case in Avro. However, what I have below completely bypasses Avro RPC classes in the avro-ipc package. Let me know if this still mandates a patch proposal on Jira. Basically - I implemented my own "Responder-Server" combination without relying on reflection based method invocation of "classic" Avro. Link to source code below:
https://github.com/openrtb/openrtb2x/blob/2.0/demand-side/dsp-core/src/main/java/org/openrtb/dsp/core/DemandSideServer.java Pankaj On Wed, Feb 27, 2013 at 4:38 PM, Pankaj Shroff <[email protected]> wrote: > Doug - thanks again. > > I agree with you. I have been looking into it for the past few days and it > seems like this will require quite a bit of refactoring. I will try to > follow up on Jira with more specific details. > > Thanks > Pankaj > > > > On Wed, Feb 27, 2013 at 2:50 PM, Doug Cutting <[email protected]> wrote: > >> Pankaj, >> >> Avro RPC is currently specified to always uses the binary encoding >> (like Avro data files). This might reasonably be extended to JSON. >> First we'd need to specify the new wire format. Probably Avro's >> framing would not make sense for JSON-encoded RPC over HTTP. Then >> we'd need to figure what of the existing Java implementation might be >> reused or adapted. At a glance, it doesn't look to me like a few >> one-line changes would suffice, adding methods where things are >> hardwired, that rather more substantial changes would be required, but >> I might be missing something. If you're interested in pursuing this >> then please file an issue in Jira where you can propose changes to the >> specification and implementation. >> >> Cheers, >> >> Doug >> >> On Wed, Feb 27, 2013 at 11:16 AM, Pankaj Shroff <[email protected]> >> wrote: >> > I guess my question is more basic - given that this is somewhat >> specific to >> > my own use case: >> > >> > How does one use other forms of Encoder/Decoder implementations that are >> > available in the Avro library along with the Avro-Ipc SDK. >> > >> > As of 1.7.3, I see that the only Encoding/Decoding that Avro-ipc >> supports is >> > the BinaryEncoding >> > >> > Pankaj >> > >> > >> > >> > On Mon, Feb 25, 2013 at 2:25 PM, Pankaj Shroff <[email protected]> >> wrote: >> >> >> >> Doug >> >> >> >> Perhaps you answered a portion of my conundrum in another thread >> >> (permalink below) - but there is still the handshake and reuse of >> invocation >> >> logic question. Let me also think about this a little bit. >> >> >> >> Thanks in any case. Avro is a great tool in any case! >> >> >> >> >> >> >> http://mail-archives.apache.org/mod_mbox/avro-user/201302.mbox/%3CCALEq1Z_rt8FasjSR%2B%2BOOgE3ogrAh0Y%2BtL3z47hznuiBAtfvWmw%40mail.gmail.com%3E >> >> >> >> >> >> Pankaj >> >> >> >> [email protected] >> >> >> >> >> >> >> >> On Mon, Feb 25, 2013 at 1:38 PM, Doug Cutting <[email protected]> >> wrote: >> >>> >> >>> This sounds like a different RPC wire format than Avro's. Avro's >> >>> Requestor and Responder implement Avro's RPC wire format. Avro's >> >>> Encode/Decoder and DatumReader/DatumWriter APIs should facilitate >> >>> implementation of other RPC wire formats that include Avro data. >> >>> Avro's Transceiver API may or may not be reusable, since it assumes >> >>> Avro-style framing. Parts of Requestor and Responder *might* be >> >>> reusable and some refactoring of those classes *might* make such reuse >> >>> easier, but there's not that much logic there that's not specific to >> >>> Avro's wire format, so it might be just as easy to reimplement this >> >>> layer for a different wire format. It's hard for me to say without >> >>> seeing a patch with a proposed refactoring. Does that make sense? >> >>> >> >>> Doug >> >>> >> >>> On Mon, Feb 25, 2013 at 9:08 AM, Pankaj Shroff <[email protected]> >> wrote: >> >>> > Hi >> >>> > >> >>> > We are using Avro for implementing an open source reference >> >>> > implementation >> >>> > of the OpenRTB protocol. >> >>> > >> >>> > We have made a design goal to model the protocol using Avro protocol >> >>> > files >> >>> > (.avpr) and generate types defined in the protocol schema using >> Avro . >> >>> > The >> >>> > challenge is that the protocol does not necessarily require the use >> of >> >>> > Avro/ >> >>> > Binary wire encoding - or even the use of Avro/ RPC context. In fact >> >>> > many >> >>> > counter parties have proprietary implementations supporting either >> >>> > Protobuf >> >>> > or Json encoding. >> >>> > >> >>> > Now, there is a Json encoder/decoder in the Avro package but it >> seems >> >>> > that >> >>> > the approach is a "schema-first" approach. The JsonEncoder assumes >> that >> >>> > the >> >>> > encoding on the wire still follows the Avro Json encoding - which >> >>> > includes a >> >>> > handshake followed by schema confirmation on both sides (client and >> >>> > server). >> >>> > >> >>> > For the protocol we are implementing - this presents 2 problems if >> >>> > Avro/ >> >>> > binary is not the chose encoding type for both sides - and if >> instead, >> >>> > lets >> >>> > say, raw Json encoding is being used >> >>> > >> >>> > 1) the handshake is rather Avro specific - and we would like to >> >>> > completely >> >>> > skip it if both sides have agreed on using raw json encoding - there >> >>> > may be >> >>> > a separate info-exchange phase in the protocol that is protocol >> >>> > specific and >> >>> > does not need to use Avro handshake. Is it possible to use Avro RPC >> >>> > without >> >>> > the handshake? >> >>> > >> >>> > 2) we would like to use the data binding and schema resolution as >> >>> > implemented by the SpecificResponder class in Avro - but extend it >> to >> >>> > use >> >>> > raw JSON - not Avro JSON - encodings. >> >>> > >> >>> > 3) We would prefer not to have to override the >> "respond(List<buffers>)" >> >>> > method of the base class Responder. This implementation always >> performs >> >>> > handshake and always uses BinaryEncoder/Decoder which removes any >> >>> > flexibility of using a different encoder /decoder in a derived >> class. >> >>> > We >> >>> > would prefer if the Responder or some derived class saves the chosen >> >>> > Decoder/ encoder as a protected property of the Responder object. >> >>> > Instead of >> >>> > instantiating BinaryEncoder/ Decoder objects on the fly within the >> >>> > respond >> >>> > method, it would be great if this was made more extensible and if >> the >> >>> > Encoder/Decoder can be specified during construction. >> >>> > >> >>> > 4) For future flexibility it would be great to have the handshake >> >>> > functionality available in sub-classes of Responder as an inherited >> >>> > method >> >>> > (instead of private scope right now). >> >>> > >> >>> > I would welcome any suggestions/corrections. >> >>> > >> >>> > Pankaj >> >>> > >> >>> > -- >> >>> > Pankaj Shroff >> >>> > twitter: @chompi >> >>> > http://github.com/chompi >> >>> > http://github.com/openrtb/openrtb2x >> >>> > >> >> >> >> >> >> >> >> >> >> -- >> >> Pankaj Shroff >> >> [email protected] >> > >> > >> > >> > >> > -- >> > Pankaj Shroff >> > [email protected] >> > > > > -- > Pankaj Shroff > [email protected] > -- Pankaj Shroff [email protected]
