This looks reasonable to me. Cheers,
Doug On Tue, Mar 12, 2013 at 2:01 PM, Pankaj Shroff <[email protected]> wrote: > Quick update on this issue. I think I finally figured out a way to bypass > Avro style "handshake" when employing "custom" format or content type > implementation but still trying to reuse or benefit from the > Serialize/Deserialize capabilities of Avro. Perhaps the following usage is > the "intended" REUSE use case in Avro. However, what I have below completely > bypasses Avro RPC classes in the avro-ipc package. Let me know if this still > mandates a patch proposal on Jira. Basically - I implemented my own > "Responder-Server" combination without relying on reflection based method > invocation of "classic" Avro. Link to source code below: > > https://github.com/openrtb/openrtb2x/blob/2.0/demand-side/dsp-core/src/main/java/org/openrtb/dsp/core/DemandSideServer.java > > Pankaj > > > > On Wed, Feb 27, 2013 at 4:38 PM, Pankaj Shroff <[email protected]> wrote: >> >> Doug - thanks again. >> >> I agree with you. I have been looking into it for the past few days and it >> seems like this will require quite a bit of refactoring. I will try to >> follow up on Jira with more specific details. >> >> Thanks >> Pankaj >> >> >> >> On Wed, Feb 27, 2013 at 2:50 PM, Doug Cutting <[email protected]> wrote: >>> >>> Pankaj, >>> >>> Avro RPC is currently specified to always uses the binary encoding >>> (like Avro data files). This might reasonably be extended to JSON. >>> First we'd need to specify the new wire format. Probably Avro's >>> framing would not make sense for JSON-encoded RPC over HTTP. Then >>> we'd need to figure what of the existing Java implementation might be >>> reused or adapted. At a glance, it doesn't look to me like a few >>> one-line changes would suffice, adding methods where things are >>> hardwired, that rather more substantial changes would be required, but >>> I might be missing something. If you're interested in pursuing this >>> then please file an issue in Jira where you can propose changes to the >>> specification and implementation. >>> >>> Cheers, >>> >>> Doug >>> >>> On Wed, Feb 27, 2013 at 11:16 AM, Pankaj Shroff <[email protected]> >>> wrote: >>> > I guess my question is more basic - given that this is somewhat >>> > specific to >>> > my own use case: >>> > >>> > How does one use other forms of Encoder/Decoder implementations that >>> > are >>> > available in the Avro library along with the Avro-Ipc SDK. >>> > >>> > As of 1.7.3, I see that the only Encoding/Decoding that Avro-ipc >>> > supports is >>> > the BinaryEncoding >>> > >>> > Pankaj >>> > >>> > >>> > >>> > On Mon, Feb 25, 2013 at 2:25 PM, Pankaj Shroff <[email protected]> >>> > wrote: >>> >> >>> >> Doug >>> >> >>> >> Perhaps you answered a portion of my conundrum in another thread >>> >> (permalink below) - but there is still the handshake and reuse of >>> >> invocation >>> >> logic question. Let me also think about this a little bit. >>> >> >>> >> Thanks in any case. Avro is a great tool in any case! >>> >> >>> >> >>> >> >>> >> http://mail-archives.apache.org/mod_mbox/avro-user/201302.mbox/%3CCALEq1Z_rt8FasjSR%2B%2BOOgE3ogrAh0Y%2BtL3z47hznuiBAtfvWmw%40mail.gmail.com%3E >>> >> >>> >> >>> >> Pankaj >>> >> >>> >> [email protected] >>> >> >>> >> >>> >> >>> >> On Mon, Feb 25, 2013 at 1:38 PM, Doug Cutting <[email protected]> >>> >> wrote: >>> >>> >>> >>> This sounds like a different RPC wire format than Avro's. Avro's >>> >>> Requestor and Responder implement Avro's RPC wire format. Avro's >>> >>> Encode/Decoder and DatumReader/DatumWriter APIs should facilitate >>> >>> implementation of other RPC wire formats that include Avro data. >>> >>> Avro's Transceiver API may or may not be reusable, since it assumes >>> >>> Avro-style framing. Parts of Requestor and Responder *might* be >>> >>> reusable and some refactoring of those classes *might* make such >>> >>> reuse >>> >>> easier, but there's not that much logic there that's not specific to >>> >>> Avro's wire format, so it might be just as easy to reimplement this >>> >>> layer for a different wire format. It's hard for me to say without >>> >>> seeing a patch with a proposed refactoring. Does that make sense? >>> >>> >>> >>> Doug >>> >>> >>> >>> On Mon, Feb 25, 2013 at 9:08 AM, Pankaj Shroff <[email protected]> >>> >>> wrote: >>> >>> > Hi >>> >>> > >>> >>> > We are using Avro for implementing an open source reference >>> >>> > implementation >>> >>> > of the OpenRTB protocol. >>> >>> > >>> >>> > We have made a design goal to model the protocol using Avro >>> >>> > protocol >>> >>> > files >>> >>> > (.avpr) and generate types defined in the protocol schema using >>> >>> > Avro . >>> >>> > The >>> >>> > challenge is that the protocol does not necessarily require the use >>> >>> > of >>> >>> > Avro/ >>> >>> > Binary wire encoding - or even the use of Avro/ RPC context. In >>> >>> > fact >>> >>> > many >>> >>> > counter parties have proprietary implementations supporting either >>> >>> > Protobuf >>> >>> > or Json encoding. >>> >>> > >>> >>> > Now, there is a Json encoder/decoder in the Avro package but it >>> >>> > seems >>> >>> > that >>> >>> > the approach is a "schema-first" approach. The JsonEncoder assumes >>> >>> > that >>> >>> > the >>> >>> > encoding on the wire still follows the Avro Json encoding - which >>> >>> > includes a >>> >>> > handshake followed by schema confirmation on both sides (client and >>> >>> > server). >>> >>> > >>> >>> > For the protocol we are implementing - this presents 2 problems if >>> >>> > Avro/ >>> >>> > binary is not the chose encoding type for both sides - and if >>> >>> > instead, >>> >>> > lets >>> >>> > say, raw Json encoding is being used >>> >>> > >>> >>> > 1) the handshake is rather Avro specific - and we would like to >>> >>> > completely >>> >>> > skip it if both sides have agreed on using raw json encoding - >>> >>> > there >>> >>> > may be >>> >>> > a separate info-exchange phase in the protocol that is protocol >>> >>> > specific and >>> >>> > does not need to use Avro handshake. Is it possible to use Avro RPC >>> >>> > without >>> >>> > the handshake? >>> >>> > >>> >>> > 2) we would like to use the data binding and schema resolution as >>> >>> > implemented by the SpecificResponder class in Avro - but extend it >>> >>> > to >>> >>> > use >>> >>> > raw JSON - not Avro JSON - encodings. >>> >>> > >>> >>> > 3) We would prefer not to have to override the >>> >>> > "respond(List<buffers>)" >>> >>> > method of the base class Responder. This implementation always >>> >>> > performs >>> >>> > handshake and always uses BinaryEncoder/Decoder which removes any >>> >>> > flexibility of using a different encoder /decoder in a derived >>> >>> > class. >>> >>> > We >>> >>> > would prefer if the Responder or some derived class saves the >>> >>> > chosen >>> >>> > Decoder/ encoder as a protected property of the Responder object. >>> >>> > Instead of >>> >>> > instantiating BinaryEncoder/ Decoder objects on the fly within the >>> >>> > respond >>> >>> > method, it would be great if this was made more extensible and if >>> >>> > the >>> >>> > Encoder/Decoder can be specified during construction. >>> >>> > >>> >>> > 4) For future flexibility it would be great to have the handshake >>> >>> > functionality available in sub-classes of Responder as an inherited >>> >>> > method >>> >>> > (instead of private scope right now). >>> >>> > >>> >>> > I would welcome any suggestions/corrections. >>> >>> > >>> >>> > Pankaj >>> >>> > >>> >>> > -- >>> >>> > Pankaj Shroff >>> >>> > twitter: @chompi >>> >>> > http://github.com/chompi >>> >>> > http://github.com/openrtb/openrtb2x >>> >>> > >>> >> >>> >> >>> >> >>> >> >>> >> -- >>> >> Pankaj Shroff >>> >> [email protected] >>> > >>> > >>> > >>> > >>> > -- >>> > Pankaj Shroff >>> > [email protected] >> >> >> >> >> -- >> Pankaj Shroff >> [email protected] > > > > > -- > Pankaj Shroff > [email protected]
