Uri, a patch of the docs/doxygen with this addition would be higly welcome!
-- i ----- Original Message ----- > As a general comment -- There's no need to manipulate the nbytes > every time you write to the output. > If you call the initial TSVConnWrite with INT64_MAX, you just need to > set the nbytes to the final amount once (So the vconn will know to > send you a WRITE_COMPLETE when it reaches this amount). > If you know the real payload size before setting up the vconn, just > call TSVConnWrite with the real size and save yourself the hassle of > adjusting the NBytes.... > Cheers, > Uri > From: [email protected] > To: [email protected] > Date: Fri, 15 Mar 2013 11:13:33 -0700 > Subject: Re: API Question > Shaun, > Thanks to your input I was able to identify the source of the > problem. > In my main handle transformation method I was declaring a local > variable int64_t nBytesWritten = 0; > Then depending on what state I was in I was calling the appropriate > handler sub method. Each of the sub methods returns the number of > bytes written to the output by that method and I was aggregating > those values via > nBytesWritten += subMethodCall(…); > At the end of my handle transformation method I was calling: > if (nBytesWritten > 0) { > /* Modify the output > * VIO to reflect how much data the output connection should > * expect. This allows the output connection to know when it > * is done reading. We then reenable the output connection so > * that it can consume the data we just gave it. > */ > TSVIONBytesSet(data->output_vio, nBytesWritten); > TSVIOReenable(data->output_vio); > } > At any rate this was working for small payloads but when the method > was re-entered nBytesWritten was being re-initialized to 0 and then > incremented. So the key is knowing that the value of nBytesWritten > must be accumulated across invocations of the transformation handler > . > by making this value part of the transaction data, I am able to > preserve it across the invocations and the modified code: > if (data->nBytesWritten > 0) { > /* Modify the output > * VIO to reflect how much data the output connection should > * expect. This allows the output connection to know when it > * is done reading. We then reenable the output connection so > * that it can consume the data we just gave it. > */ > TSVIONBytesSet(data->output_vio, data->nBytesWritten); > TSVIOReenable(data->output_vio); > } > I am worried though about what you said regarding it getting confused > and sending a WRITE_COMPLETE event pre-maturely. Right now with my > current test cases seem to be working as expected but then again, it > all seems like voodoo and black magic under the ATS hood and I don't > know that there isn't a test case which would break this. I wish I > could be more confident that the code I am writing is going to work > as intended under all cases. > For the moment however everything s eems to work. I am logging the > values of the output nbytes and ndone. And it would seem that ndone > is not getting incremented at all during the multiple invocations of > my transform handler. Ideally what I would like to do is to have the > down stream output handler consume data as soon as it gets it from > my transform, but not to stop or terminate until I have finished > sending ALL of my data downstream but I am afraid to breathe on the > code at this point until I find a breaking case which will mandate a > change. > Thank you for your help, you have truly made my day. > Steve Owens > From: Shaun mcginnity < [email protected] > > Reply-To: " [email protected] " < > [email protected] > > Date: Fri, 15 Mar 2013 10:35:01 -0700 > To: " [email protected] " < > [email protected] > > Subject: Re: API Question > Yes, it allows the VIO to know when it is done. We only set it when > we have read all the data from the input. I suspect if you > continually set it the VIO might think it is done and send a > WRITE_COMPLETE but you actually still have more to write. > Shaun > On Fri, Mar 15, 2013 at 5:20 PM, Owens, Steve < > [email protected] > wrote: > > Shaun, > > > Looking at the code for TSVIONBytesSet we have > > > void 281 TSVIONBytesSet ( TSVIO viop , int64_t nbytes ) 282 { 283 > > sdk_assert ( sdk_sanity_check_iocore_structure ( viop ) == > > TS_SUCCESS ); 284 sdk_assert ( nbytes >= 0); 285 286 VIO * vio = ( > > VIO *) viop ; 287 vio -> nbytes = nbytes ; 288 } > > > It looks like this method updates the nbytes of the VIO. But it > > doesn't explain what the implications of changing this value are. > > I > > am presuming that when I called TSVConnWrite(output_conn, contp, > > data->output_reader, INT64_MAX); That the value of vio->nbytes got > > set to INT64_MAX. Since my transform may or may not know how much > > upstream data it needs to process I don't see any way for it to > > update this value to anything meaningful and it makes me wonder why > > I would want to. > > > In my STATE_EXIT processing I am calling: > > TSContCall(TSVIOContGet(input_vio), TS_EVENT_VCONN_WRITE_COMPLETE, > > input_vio); But I don't call: > > TSVConnShutdown(TSTransformOutputVConnGet(contp), 0, 1); until I > > receive the TS_EVENT_VCONN_WRITE_COMPLETE event in my > > callbackHandler. > > > What would happen if I continuously update > > TSVIONBytesSet(output_vio, > > total_written) to indicate how much I have written thus far? > > > Steve Owens > > > From: Shaun mcginnity < [email protected] > > > > Reply-To: " [email protected] " < > > [email protected] > > > > Date: Fri, 15 Mar 2013 09:48:05 -0700 > > > To: " [email protected] " < > > [email protected] > > > > Subject: Re: API Question > > > Hi Steve, > > > I believe that in a transform you should only call TSVConnWrite > > once. > > > You say you are shutting down the VIOs. Are you doing this when you > > receive a WRITE_COMPLETE event? If not you may be closing the VIO > > early. You can tell the output VIO that all the data has been > > written to it using TSVIONBytesSet(output_vio, total_written); > > > Regards, > > > Shaun > > > On Fri, Mar 15, 2013 at 4:25 PM, Owens, Steve < > > [email protected] > wrote: > > > > Shaun, > > > > > > This is extremely helpful. In fact it would make a great addition > > > to > > > the traffic server on line documentation. Personally I prefer to > > > understand what the methods I am calling do rather than not. > > > > > > The one question that remains unanswered is whether or not this > > > method needs to be called more than once. > > > > > > The reason for my question is this: > > > > > > I have a transform which essentially prepends and postpends > > > content > > > to the upstream content being served. It works in states > > > > > > STATE_START = write the prefix content downstream and shift to > > > STATE_MIDDLE. > > > > > > STATE_MIDDLE = copy upstream content to downstream until there is > > > no > > > more content and then shift to STATE_END. > > > > > > STATE_END = write the suffix content downstream and shift to > > > STATE_EXIT. > > > > > > STATE_EXIT = clean up the transaction and shut down the VIO's > > > because > > > were done. > > > > > > This plugin works with smaller upstream content as expected. > > > However > > > when the upstream content gets larger than about 30K, what I am > > > seeing is that from the transform plugin's perspective it is > > > working > > > as expected. The expected amount of data is being copied > > > downstream > > > and the suffix is being written. I can tell this because I am > > > logging the values of upstream_avail, upstream_todo, bytesWritten > > > etc. And everything adds up to all content being delivered. > > > However > > > the client is receiving truncated content and I am baffled as to > > > why > > > this my be happening. I thought that maybe I was not properly > > > using > > > the TSVConnWrite method but from your explanation of the method > > > below, it would seem that this is possibly not the case. > > > > > > At any rate, > > > > > > Thank you for the explanation. > > > > > > Best Regards, > > > > > > Steve Owens > > > > > > From: Shaun mcginnity < [email protected] > > > > > > > Reply-To: " [email protected] " < > > > [email protected] > > > > > > > Date: Fri, 15 Mar 2013 07:09:12 -0700 > > > > > > To: " [email protected] " < > > > [email protected] > > > > > > > Subject: Re: API Question > > > > > > Hi Steve, > > > > > > TSVConnWrite initiates a write operation on the transform's > > > downstream VConnection. The last parameter is the total amount of > > > data you are going to write. INT64_MAX is effectively unlimited. > > > > > > When you have data to write you get the VIO's buffer > > > (TSVIOBufferGet(output_vio)), and copy the data into it. Then the > > > VIO is reenabled. Reenabling the VIO causes an IMMEDIATE event to > > > be > > > sent to the VIO's VConnection. When the VConnection's event > > > handler > > > has finished with the data it will call your continuation back. > > > If > > > it expects you to send more data it will pass a WRITE_READY > > > event. > > > If you have given the VIO the total amount of data as set in > > > TSVConnWrite it will call you back with a WRITE_COMPLETE event. > > > Thisbehaviour is the same as what the transform should do to its > > > input VIO. > > > > > > Since the downstream VConnection is the client and / or cache, if > > > you > > > do a finite write, i.e. not INT64_MAX, then the HTTP transaction > > > will close when the limit is reached. Issuing another write in > > > this > > > case will probably not work. You can do multiple writes on a > > > NetVConnection though, e.g. as provided by TSNetConnect. > > > > > > Hope that helps. > > > > > > Shaun > > > > > > -----Original Message----- > > > > > > From: James Peach [ mailto:[email protected] ] > > > > > > Sent: Friday, March 15, 2013 3:12 AM > > > > > > To: [email protected] > > > > > > Subject: Re: API Question > > > > > > On 14/03/2013, at 5:13 PM, "Owens, Steve" < > > > [email protected] > > > > > > > wrote: > > > > > > > In looking at the following documentation: > > > > http://trafficserver.apache.org/docs/trunk/sdk/io-guide/transformations.en.html > > > > > > > > > > > > > > It would seem that the only mention of TSVConnWrite is to a > > > > single > > > > parameter method. > > > > > > > > > > > > > > Yet in my plugin I am using > > > > > > > > > > > > > > data->output_vio = TSVConnWrite(output_conn, contp, > > > > data->output_reader, > > > > > > > INT64_MAX); > > > > > > > > > > > > > > What has changed about this method? > > > > > > > > > > > > > > What does it actually do? > > > > > > It schedules data to be written to the VConnection. > > > > > > > > > > > > > > Do I need to call it every time the plugin is re-enabled? Or > > > > can > > > > I > > > > just call it once? > > > > > > I don't know that I really understand the question, but if the > > > length > > > is INT64_MAX you would only call it once. If you want you cancall > > > it > > > a multiple times with shorter lengths. > > > > > > J > > > -- Igor Galić Tel: +43 (0) 664 886 22 883 Mail: [email protected] URL: http://brainsware.org/ GPG: 6880 4155 74BD FD7C B515 2EA5 4B1D 9E08 A097 C9AE
