[...] > We may all be agreeing here. All that we can do at this point is just move > forwards. If some particular change or line of development seems to be > moving the library "too far" from its original charter, then we can discuss > it then. *if* it even comes up :-)
Agreed. > > I'll agree that we can mix sync/async here (will the distinction be > > fragile?), > > I'm not sure what you mean by "fragile" here. Do you mean in the sense that > we can't mix the modes easily? Or that you must set it up "just so" do do > one mode, yet "like this" for the other mode? And that it won't be obvious > which, so you'd usually end up with a broken setup? I'm not really sure. But those are good things to keep in mind when we start designing these interfaces. > Personally, I think we can have an API that easily flows and can handle the > different modes. > > > but I'm not quire sure I understand how this will work internally to the > > filters. I think one of the bigger hurdles in handling push vs. pull on > > input filters is where the buffers come from. I'll have to think about this > > some more though... > > See one of my other posts, where I talk about turning a push filter into a > pull filter. Essentially, a person should always write their filter as a > "push" filter (like the output filters in httpd). If we want to run the > filter in a "pull" mode (something wants to pull data through a filter > chain), then we have a way to "flip" the chain and push data through in a > way that ends up as a "pull". > > Let's say that you want to operate the response stack in a pull mode (the > app is saying "give me more of the response"). The app calls > serf_get_response() to get a brigade with more of the response. Inside that > function, the network code creates a brigade with some socket data. That > data is then pushed through the response filters. Whatever pops out the > other side is then returned from serf_get_response(). > > We may have some issues with limiting the amount of data which is processed. > For example, if the app requests 200 bytes, but after pushing data through > the response stack, we end up with 300 bytes... we can easily save aside the > extra 100 bytes. But what happens if that happens *internally* to the stack? > > A bit of thought needed there. > > [ in particular, if F1 generates more data than F2 needs/wants, then should > F2 set it aside? what if F2 is tossed and F3 is brought in, and should > have received that data? ] Yes. These seem like some of the same issues that have come up lately on httpd. I think the problem in semantics here comes from the fact that all filters are being forced to define their data requirements in terms of bytes, when that isn't really what unit of data they deal with internally. A Dechunk filter only cares about chunks and chunk tags, for example. ... > > Also, since some filters may be provided > > by our library (chunk, ssl, basic auth, various header manipulations, etc) > > and some will be provided by the application (flood will have a > > content-length > > verification filter, a webservices thingy could render some xml, etc), and > > Yes, although auth isn't really a filter. There is a lot of high-level > processing related to auth (think about failing auth, asking the app for > auth data, then re-submitting the request; a filter can't do that). I don't follow you. If a filter detects that auth fails, it passes a poison pill down the filter chain. I don't know what you mean by asking the app for auth data, the app should have inserted a filter to do this, no? > > since it seems that most of the power of our library will be in the filters, > > I don't see why the application will care if it's push or pull. > > The app will have data pushed at it, or the app will pull data from the > library. Transparent to the app, some filters will execute in there. When I consider the app, I'm picturing it as a control mechanism for this filter chain, where it has already inserted strategic filters of its own into the chain and then triggered a push or pull on the chain. I don't see data spilling out the end of the chain, but I may not be seeing the whole picture yet. > >... > > That brings up a good point. We have an opportunity to come up with a > > scheme for the ordering of filters that could potentially be more flexible > > than the one for HTTP. (I think they are under more constraints). > > Suggestions? > > Filters should operate independent of what comes before/after themselves. > That means that a higher-level entity is responsible for ordering them > properly since a filter can't examine its neighbors and place itself > accordingly. > > That said: even though a filter can operate "anywhere", it can also describe > itself to some extent to the library/app about where it "should" go. This is > the model used by httpd (filter type). However, httpd has a lot of extra goo > for adding/removing those things. I don't know how much of that is needed. I have some thoughts about this in the design drifting on the flotsam in my head, but the details are not yet clear. I think the main things we have here are: 1) the app controls how filters are ordered in the chain, 2) we can create utilities that the app can use to order the chain (these utils can be complex enough so as to completely hide the ordering), 3) I completely agree that filters should know nothing about the filters before or after it in the chain. -aaron
