re: "and I think it's also a little more experimental " I meant to say: and I think it has a more realistic function
2013/1/25 Otto van der Schaaf <[email protected]>: > I hope that works out, might be helpful and prevent other from having > to go down the same road :-) I would be happy to help out with that. > > Regarding INKVIONBytesGet: > > "Gets the number of bytes to be performed by the IO operation > described by viop. This is the nbytes parameter that's passed to > INKVConnRead or INKVConnWrite." > > So, no, I don't think that modifying anything. If I remember > correctly, that call returns the total number of bytes that the will > be written to the passed in vio. So in the context of the picture > above, you will write as much bytes to the output as you get from the > input I guess. You can specify INT64_MAX if you don't know yet how > much bytes you will write, in which case trafficserver will either > send a connection:close header or send a transfer-encoding chunked. > > You might want to have a look at the gzip plugin in /experimental: I > made an effort to make to code a little more understandable then the > null-transform, and I think it's also a little more experimental :-) > > > Regards, > > Otto > > > 2013/1/25 Owens, Steve <[email protected]>: >> Otto, >> >> I have started mapping code examples from the null transform example to >> the diagram to try to connect the dots see attached. >> >> Steve Owens >> >> >> >> On 1/25/13 11:44 AM, "Owens, Steve" <[email protected]> wrote: >> >>>Otto, yes I have seen them and read them several times. The second link >>>explains in rather abstract terms what VIO's do and how they relate to >>>buffers. >>> >>>But I personally could use a little more amplification on what the >>>specific methods are that manipulate these structures and what effect that >>>calls to these methods have on the structures depicted in the images. >>> >>>The null transform plugin re-uses variables and takes enough shortcuts >>>that what it is doing is seems obscured and I am trying to pierce that >>>obscurity. I will continue to try to pierce the veil but after having >>>done so I would love to write up additional info to make it clearer to >>>dunderheads such as myself what these methods actually do in order to >>>reduce the slope of the learning curve for this technology. >>> >>>Thank you for re-directing me to these I am sure that given enough time >>>and patience I will crack the code. >>> >>> >>> >>>On 1/25/13 11:25 AM, "Otto van der Schaaf" <[email protected]> wrote: >>> >>>>Hi Steve, >>>> >>>>Did you get a chance already to have a look at >>>>http://trafficserver.apache.org/docs/v2/sdk/images/vconnection.jpg ? >>>>That image is from >>>>http://trafficserver.apache.org/docs/v2/sdk/HTTPTransformationPlugins.htm >>>>l >>>>#VIOs >>>>Both helped my understanding of the transformation api's >>>> >>>>Regards, >>>> >>>>Otto >>>> >>>> >>>>2013/1/25 Owens, Steve <[email protected]>: >>>>> One step closer >>>>> >>>>> VIO * >>>>> >>>>> PluginVC::do_io_write(Continuation * c, int64_t nbytes, IOBufferReader >>>>>* >>>>> abuffer, bool owner) >>>>> >>>>> { >>>>> >>>>> >>>>> ink_assert(!closed); >>>>> >>>>> ink_assert(magic == PLUGIN_VC_MAGIC_ALIVE); >>>>> >>>>> >>>>> if (abuffer) { >>>>> >>>>> ink_assert(!owner); >>>>> >>>>> write_state.vio.buffer.reader_for(abuffer); >>>>> >>>>> } else { >>>>> >>>>> write_state.vio.buffer.clear(); >>>>> >>>>> } >>>>> >>>>> >>>>> // Note: we set vio.op last because process_write_side looks at it to >>>>> >>>>> // tell if the VConnection is active. >>>>> >>>>> write_state.vio.mutex = c->mutex; >>>>> >>>>> write_state.vio._cont = c; >>>>> >>>>> write_state.vio.nbytes = nbytes; >>>>> >>>>> write_state.vio.ndone = 0; >>>>> >>>>> write_state.vio.vc_server = (VConnection *) this; >>>>> >>>>> write_state.vio.op = VIO::WRITE; >>>>> >>>>> >>>>> Debug("pvc", "[%u] %s: do_io_write for %"PRId64" bytes", PVC_ID, >>>>>PVC_TYPE, >>>>> nbytes); >>>>> >>>>> >>>>> // Since reentrant callbacks are not allowed on from do_io >>>>> >>>>> // functions schedule ourselves get on a different stack >>>>> >>>>> need_write_process = true; >>>>> >>>>> setup_event_cb(0, &sm_lock_retry_event); >>>>> >>>>> >>>>> return &write_state.vio; >>>>> >>>>> } >>>>> >>>>> >>>>> It would appear that the value passed in to the third argument of >>>>> TSVConnWrite ends up being assigned to write_state.vio.nbytes >>>>> >>>>> the docs on write_state.vio.nbytes state: >>>>> >>>>> /** >>>>> >>>>> Number of bytes to be done for this operation. >>>>> >>>>> >>>>> The total number of bytes this operation must complete. >>>>> >>>>> >>>>> */ >>>>> >>>>> int64_t nbytes; >>>>> >>>>> >>>>> So far clear as mud. Apologies in advance for my mental density. >>>>> >>>>> >>>>> >>>>> From: "Owens, Steve" <[email protected]> >>>>> Reply-To: "[email protected]" >>>>><[email protected]> >>>>> Date: Fri, 25 Jan 2013 11:09:08 -0800 >>>>> To: "[email protected]" <[email protected]> >>>>> Subject: Re: Api Questions >>>>> >>>>> An addendum to the previous message I have resorted to reading the >>>>>traffic >>>>> server code in order to try to discern what it does: >>>>> >>>>> I am examining >>>>> >>>>> TSVIO >>>>> >>>>> TSVConnWrite(TSVConn connp, TSCont contp, TSIOBufferReader readerp, >>>>>int64_t >>>>> nbytes) >>>>> >>>>> { >>>>> >>>>> sdk_assert(sdk_sanity_check_iocore_structure(connp) == TS_SUCCESS); >>>>> >>>>> sdk_assert(sdk_sanity_check_iocore_structure(contp) == TS_SUCCESS); >>>>> >>>>> sdk_assert(sdk_sanity_check_iocore_structure(readerp) == TS_SUCCESS); >>>>> >>>>> sdk_assert(nbytes >= 0); >>>>> >>>>> >>>>> FORCE_PLUGIN_MUTEX(contp); >>>>> >>>>> VConnection *vc = (VConnection *) connp; >>>>> >>>>> >>>>> return reinterpret_cast<TSVIO>(vc->do_io_write((INKContInternal *) >>>>>contp, >>>>> nbytes, (IOBufferReader *) readerp)); >>>>> >>>>> } >>>>> >>>>> >>>>> It would appear that the answer to what this method does lies in what >>>>> happens in the function "vc->do_io_write" yet when I search the >>>>> trafficserver 3.2.0 codebase for any file that actually contains an >>>>> implementation of "vc->do_io_write" I find that "do_io_write" is >>>>>declared in >>>>> several .h files but not defined in any .c or .cpp files. >>>>> >>>>> >>>>> Does anyone know where in the code base the implementation of >>>>>"do_io_write" >>>>> resides? >>>>> >>>>> >>>>> >>>>> >>>>> From: "Owens, Steve" <[email protected]> >>>>> Reply-To: "[email protected]" >>>>><[email protected]> >>>>> Date: Fri, 25 Jan 2013 10:48:11 -0800 >>>>> To: "[email protected]" <[email protected]> >>>>> Subject: Api Questions >>>>> >>>>> I am trying to understand the relationship between the following >>>>>functions >>>>> as they are used within a transformation plugin and so far no matter >>>>>how >>>>> hard I dig the mystery remains. I am hoping that people who have a >>>>>better >>>>> understanding of ATS than I do would be willing to help clear up any >>>>> misunderstandings I have. >>>>> >>>>> Here are the functions I find confusing: >>>>> >>>>> TSVIONBytesSet >>>>> >>>>> (TSIOBufferCopy and or TSIOBufferWrite) >>>>> >>>>> And finally >>>>> >>>>> TSVConnWrite >>>>> >>>>> >>>>> Each of the above functions takes an int_64 argument at the end to >>>>>indicate >>>>> the number of bytes but I don't fully understand the purpose of this >>>>> parameter with respect to each of the functions. >>>>> >>>>> >>>>> Suppose I have a transformation which is intended to write a prefix >>>>>spew the >>>>> response and then write a suffix to the down stream I would imagine it >>>>>looks >>>>> something like this: >>>>> >>>>> >>>>> >>>>> >>>>> void handleTransformOperation(TSCont contp) { >>>>> >>>>> TSIOBuffer buf_test; >>>>> >>>>> int_64 written = 0; >>>>> >>>>> TransormationData* pData = (TransformationData*) TSContDataGet(contp); >>>>> >>>>> TSVIO input_vio = TSTransformOutputVConnGet(contp); >>>>> >>>>> >>>>> if(data->state == STATE_BEGIN) { >>>>> >>>>> // Initialize the output_buffer and output_reader >>>>> >>>>> data->output_buffer = TSIOBufferCreate(); >>>>> >>>>> data->output_reader = TSIOBufferReaderAlloc(data->output_buffer); >>>>> >>>>> >>>>> >>>>> // Write out a prefix >>>>> >>>>> written += TSIOBufferWrite(data->output_buffer, "{Hello I am a >>>>>prefix}", >>>>> strlen(prefix) * sizeof(char)); >>>>> >>>>> data->state == STATE_MIDDLE; >>>>> >>>>> >>>>> // Figure out if the upstream wants to stop receiving WRITEREADY and >>>>> WRITE_COMPLETE events >>>>> >>>>> buf_test = TSVIOBufferGet(input_vio); >>>>> >>>>> if (!buf_test) { >>>>> >>>>> int64_t doneget = TSVIONDoneGet(input_vio); >>>>> >>>>> TSVIONBytesSet(data->output_vio, /*What goes here doneget or >>>>>written >>>>> + doneget or just written? Bonus question: why? /* ???); >>>>> >>>>> data->state = STATE_END; >>>>> >>>>> } else >>>>> >>>>> data->state = STATE_MIDDLE; >>>>> >>>>> } >>>>> >>>>> >>>>> // Notice that the above if falls through to the next state so it is >>>>> possible that the following code may or may not >>>>> >>>>> // get executed based on the buf_test result above. >>>>> >>>>> if(data->state == STATE_MIDDLE) { >>>>> >>>>> // Figure out if the upstream wants to stop receiving WRITEREADY and >>>>> WRITE_COMPLETE events >>>>> >>>>> buf_test = TSVIOBufferGet(input_vio); >>>>> >>>>> if (!buf_test) { >>>>> >>>>> int64_t doneget = TSVIONDoneGet(input_vio); >>>>> >>>>> TSVIONBytesSet(data->output_vio, /*What goes here doneget or >>>>>written >>>>> + doneget or just written? Bonus question: why? /* ???); >>>>> >>>>> data->state = STATE_END; >>>>> >>>>> } else { >>>>> >>>>> // Figure out how much data is left to read from the upstream >>>>> >>>>> int64_t upstreamBytesRemaining = TSVIONTodoGet(input_vio); >>>>> >>>>> int64_t upstreamBytesAvail = >>>>> TSIOBufferReaderAvail(TSVIOReaderGet(input_vio)); >>>>> >>>>> if(upstreamBytesRemaining > 0) { >>>>> >>>>> int64_t bytesToWrite = upstreamBytesRemaining; >>>>> >>>>> if(bytesToWrite > upstreamBytesAvail) >>>>> >>>>> bytesToWrite = upstreamBytesAvail; >>>>> >>>>> >>>>> >>>>> /* Copy the data from the read buffer to the output buffer. >>>>>*/ >>>>> >>>>> TSIOBufferCopy(TSVIOBufferGet(data->output_vio), >>>>> TSVIOReaderGet(input_vio), bytesToWrite, 0); >>>>> >>>>> >>>>> /* Tell the read buffer that we have read the data and are >>>>>no >>>>> >>>>> * longer interested in it. >>>>> >>>>> */ >>>>> >>>>> TSIOBufferReaderConsume(TSVIOReaderGet(input_vio), bytesToWrite); >>>>> >>>>> >>>>> /* Modify the input VIO to reflect how much data we've >>>>> >>>>> * completed. >>>>> >>>>> */ >>>>> >>>>> int64_t doneget = TSVIONDoneGet(input_vio); >>>>> >>>>> TSVIONDoneSet(input_vio, /* This is where things get really muddy >>>>>what >>>>> goes here? doneget or doneget + bytesToWrite or doneget + bytesToWrite >>>>>+ >>>>> written */ ??); >>>>> >>>>> /* What is the impact of calling this later in the handler >>>>>if it >>>>> has already been called here? >>>>> >>>>> } >>>>> >>>>> >>>>> /* Now we check the input VIO to see if there is data left to >>>>> >>>>> * read. >>>>> >>>>> */ >>>>> >>>>> upstreamBytesRemaining = TSVIONTodoGet(input_vio); >>>>> >>>>> if (upstreamBytesRemaining > 0) { >>>>> >>>>> if (upstreamBytesAvail > 0) { >>>>> >>>>> /* If there is data left to read, then we reenable the output >>>>> >>>>> * connection by reenabling the output VIO. This will wake up >>>>> >>>>> * the output connection and allow it to consume data from the >>>>> >>>>> * output buffer. >>>>> >>>>> */ >>>>> >>>>> TSVIOReenable(data->output_vio); >>>>> >>>>> >>>>> /* Call back the input VIO continuation to let it know that we >>>>> >>>>> * are ready for more data. >>>>> >>>>> */ >>>>> >>>>> TSContCall(TSVIOContGet(input_vio), >>>>> >>>>> TS_EVENT_VCONN_WRITE_READY, input_vio); >>>>> >>>>> } >>>>> >>>>> } else { >>>>> >>>>> /* There is no data left to read so set state to STATE_END and */ >>>>> >>>>> data->state = STATE_END; >>>>> >>>>> } >>>>> >>>>> } >>>>> >>>>> >>>>> // The above can fall through to this if block >>>>> >>>>> if(data->state == STATE_END) { >>>>> >>>>> /* Spew a suffix */ >>>>> >>>>> const char* suffix = "{i am a suffix}"; >>>>> >>>>> int64_t localWritten = TSIOBufferWrite(data->output_buffer, >>>>>suffix, >>>>> (strlen(suffix) - 1) * sizeof(char)); >>>>> >>>>> >>>>> >>>>> int64_t doneget = TSVIONDoneGet(input_vio); >>>>> >>>>> TSVIONBytesSet(data->output_vio, /* What goes here? Why? */??); >>>>> >>>>> TSVIOReenable(data->output_vio); >>>>> >>>>> >>>>> /* Call back the input VIO continuation to let it know that we >>>>> >>>>> * have completed the write operation. >>>>> >>>>> */ >>>>> >>>>> TSContCall(TSVIOContGet(input_vio), >>>>>TS_EVENT_VCONN_WRITE_COMPLETE, >>>>> input_vio); >>>>> >>>>> } >>>>> >>>>> >>>>> /* I am presuming we only want to call this once per invocation of >>>>> handlerOperation But I am not sure */ >>>>> >>>>> data->output_vio = TSVConnWrite(output_conn, contp, >>>>>data->output_reader, >>>>> /* What to put here hmmm and why? */ ?? ); >>>>> >>>>> } >>>>> >>>>> >>>>> >>>>> I really want to understand how these methods relate and what they do. >>>>>I >>>>> would be willing to help update/contribute to the ATS docs but first I >>>>>would >>>>> need to understand what they do and so far the examples that are >>>>>available >>>>> have not made it clear to me what is going on as yet so any help would >>>>>be >>>>> appreciated. >>>>> >>>>> >>>>> >>>>> >>> >>
