Is anybody still willing to work on this with me?
On Fri, Jan 24, 2014 at 12:57 PM, Kenneth Adam Miller < [email protected]> wrote: > In that case, it would be a separately named library; the developers of > their application wouldn't execute their application in a pintool context, > so they would use the vanilla library, because they would never be able to > get their application off the ground due to link errors from undefined > symbols and runtime unresolved references. They would never think to > themselves "I'm going to run my normal everyday application which uses > regular threads in a regular environment with an obscure variant of > zeromq." As a result, the vanilla zeromq would spawn threads the normal > way, and hence they would get instrumented. Sure you would have some > unnecessary data/space consumption from replicated > > This particular library could just be named to something like libpinzmq or > libzmq-pintool. Then it should work fine. > > > On Fri, Jan 24, 2014 at 12:51 PM, Lindley French <[email protected]>wrote: > >> Ideally, it would be possible to use ZMQ simultaneously inside of and >> outside of a pintool context. I'm not familiar with pintools, but if it's >> an instrumentation framework, what if they code you're trying to instrument >> *also* uses ZMQ? >> >> >> On Fri, Jan 24, 2014 at 1:44 PM, Kenneth Adam Miller < >> [email protected]> wrote: >> >>> No; It calls PIN_SpawnInternalThread, which only makes sense in a >>> pintool's executing environment. Basically, if you use this particular >>> modified version of zmq, you specifically want to use zmq in a pintool >>> context. >>> >>> I wouldn't say my changes are "harm", they are what I see as necessary >>> in order to realize pintool utilization of zeromq. I'm just not as >>> familiar with the zeromq code base as perhaps you guys are. :-) >>> >>> >>> On Fri, Jan 24, 2014 at 12:42 PM, Lindley French <[email protected]>wrote: >>> >>>> Question: Does your modified code work properly outside of a pintool >>>> context? EG, does it at least pass the "do no harm" test, before you worry >>>> about whether it does what you want it to do? >>>> >>>> >>>> On Fri, Jan 24, 2014 at 1:37 PM, Kenneth Adam Miller < >>>> [email protected]> wrote: >>>> >>>>> It must use pintools, because I had to refactor libzmq in order to >>>>> play nice with pintool; that's what the question is about. The 200 line >>>>> program perfectly replicates the issue of not being able to create a >>>>> router-proxy-dealer setup exactly as laid out in the manual; I get the >>>>> same >>>>> error using that sample as in my currently closed source pintool. >>>>> >>>>> The reason why I have to use a pintool to demonstrate the problem is >>>>> because I must give an example that shows the augmented libzmq operating, >>>>> and the augmented libzmq spawns threads using the PIN thread API (just as >>>>> the pintool manual specifies that thread control must in a pintool >>>>> executing environment). I just realized I didn't try to explain that >>>>> thread >>>>> spawning and thread control primitives must be done through the PIN api; >>>>> basically because PIN is a dynamic instrumentation framework, if you spawn >>>>> a thread, even in the instrumentation or analysis routines that you >>>>> define, >>>>> using the normal facilities, then PIN will have a recursive condition >>>>> against the thread that you innocently meant to spawn to help you do >>>>> analysis or instrumentation faster. In order to prevent this, they provide >>>>> the PIN thread APIs. >>>>> >>>>> I like zeromq, and I would like to use it to coordinate my analysis >>>>> and instrumentation routines. The problem is, the zeromq library uses some >>>>> thread control and primitives that it shouldn't. >>>>> >>>>> I'm sorry, but I can't make the example any less complicated by moving >>>>> the example out of a pintool context. I can try and eliminate some code >>>>> lines... If you have any questions about pintool itself, please feel free >>>>> to ask. I'm the one getting help from you guys in the long run, so I'm >>>>> more >>>>> than happy to help you help me :-) >>>>> >>>>> To be honest, I acknowledge that I'm bringing in a complex library >>>>> that has nothing to do with this mailing list itself, but this porting >>>>> needs to occur in order for the pintool community to benefit from the >>>>> capabilities that zmq offers. I'm highly motivated by zeromq's success and >>>>> flexibility, enough that I'm willing to dive an into unfamiliar code dig >>>>> through and find what I think needs to be changed and change it. >>>>> >>>>> >>>>> On Fri, Jan 24, 2014 at 12:02 PM, Pieter Hintjens <[email protected]>wrote: >>>>> >>>>>> I saw the repository and saw some code but a 200+ line program isn't >>>>>> really a minimal test case. As far as I understand, using pintools >>>>>> should be transparent to users, since it doesn't modify the ZMQ API, >>>>>> right? So you can make a minimal test case that doesn't require any >>>>>> knowledge of pintools to understand. >>>>>> >>>>>> On Fri, Jan 24, 2014 at 8:39 AM, Kenneth Adam Miller >>>>>> <[email protected]> wrote: >>>>>> > The example is on the github repo where it has >>>>>> PinTool-ErrorReplication. >>>>>> > >>>>>> > >>>>>> > On Fri, Jan 24, 2014 at 8:29 AM, Pieter Hintjens <[email protected]> >>>>>> wrote: >>>>>> >> >>>>>> >> Hi Kenneth, >>>>>> >> >>>>>> >> Could you post a minimal example of the code that causes an error? >>>>>> If >>>>>> >> possible, in C, using the low level API. First step is to let us >>>>>> >> simply inspect this and check for obvious misuse of sockets. >>>>>> >> >>>>>> >> The "socket operation on non-socket" can happen when you pass an >>>>>> >> invalid argument to a zmq socket call, or when you are trying to >>>>>> use a >>>>>> >> closed socket. >>>>>> >> >>>>>> >> -Pieter >>>>>> >> >>>>>> >> On Fri, Jan 24, 2014 at 7:28 AM, Kenneth Adam Miller >>>>>> >> <[email protected]> wrote: >>>>>> >> > I would like to use zeromq to work with intel pintool ( >>>>>> pintool.org). >>>>>> >> > Pintool >>>>>> >> > is a dynamic binary instrumentation framework; it allows the >>>>>> developer >>>>>> >> > to >>>>>> >> > specify some callbacks at many levels of execution of any binary >>>>>> >> > program. >>>>>> >> > But to allow zeromq to work with it, you have to restrict thread >>>>>> >> > creation >>>>>> >> > down to only the context or whatever other threads you have, and >>>>>> then >>>>>> >> > specify that the thread creation and any other thread apis go >>>>>> through >>>>>> >> > the >>>>>> >> > pin interface. If they don't then you potentially have a >>>>>> recursive >>>>>> >> > instrument and execute condition, where a thread you want to >>>>>> work for >>>>>> >> > the >>>>>> >> > instrumentation/analysis side is also being targeted for >>>>>> >> > instrumentation. >>>>>> >> > >>>>>> >> > My changes to zeromq consist of some minor alterations to >>>>>> thread.hpp and >>>>>> >> > mutex.hpp. You can see what I've changed on my forked repository >>>>>> over >>>>>> >> > here-all I do is replace the threading primitives that each >>>>>> class is >>>>>> >> > abstracting from with an instance for pin's. >>>>>> >> > >>>>>> >> > I know for certain that my changes are at taking effect, because >>>>>> what I >>>>>> >> > can >>>>>> >> > see is that inter thread communication for just ZMQ_PULL and >>>>>> ZMQ_PUSH >>>>>> >> > works >>>>>> >> > just fine. But when I go to create dealer's and router's and >>>>>> link them >>>>>> >> > with >>>>>> >> > a proxy I start getting errors with code that came straight from >>>>>> the >>>>>> >> > manual, >>>>>> >> > which is really not good. I would prefer that my hack continue >>>>>> to work, >>>>>> >> > and >>>>>> >> > I have confidence that with some help I can make it work. >>>>>> >> > >>>>>> >> > custom libzmq implementation: >>>>>> >> > https://github.com/KennethAdamMiller/libzmq >>>>>> >> > >>>>>> >> > ./autogen.sh >>>>>> >> > >>>>>> >> > (for 64 bit) >>>>>> >> > ./configure CXXFLAGS="-DBIGARRAY_MULTIPLIER=1 -Wall -Werror >>>>>> >> > -Wno-unknown-pragmas -fno-stack-protector -DTARGET_IA32E >>>>>> -DHOST_IA32E >>>>>> >> > -fPIC >>>>>> >> > -DTARGET_LINUX -I$HOME/pin-2.13/source/include/pin >>>>>> >> > -I$HOME/pin-2.13/source/include/pin/gen >>>>>> >> > -I$HOME/pin-2.13/extras/components/include >>>>>> >> > -I$HOME/pin-2.13/extras/xed2-intel64/include >>>>>> >> > -I$HOME/pin-2.13/source/tools/InstLib -O3 -fomit-frame-pointer >>>>>> >> > -fno-strict-aliasing -g -std=c++11 -L$HOMEpin-2.13/intel64/lib >>>>>> >> > -L$HOME/pin-2.13/intel64/lib-ext >>>>>> -L$HOME/pin-2.13/intel64/runtime/glibc >>>>>> >> > -L$HOME/pin-2.13/extras/xed2-intel64/lib -lpin -lxed -ldwarf >>>>>> -lelf -ldl >>>>>> >> > " >>>>>> >> > >>>>>> >> > sudo make install >>>>>> >> > >>>>>> >> > repeatable pintool error: >>>>>> >> > https://github.com/KennethAdamMiller/PinTool-ErrorReplication >>>>>> >> > Building this pintool is easy: >>>>>> >> > download a pin kit and extract to some location, like ~/pin-2.13 >>>>>> >> > >>>>>> >> > export PIN_ROOT=location >>>>>> >> > export TOOLS_ROOT=$PIN_ROOT/source/tools >>>>>> >> > export CONFIG_ROOT=$TOOLS_ROOT/Config >>>>>> >> > >>>>>> >> > then just cd to where you checked out my tool and type make. >>>>>> You'll see >>>>>> >> > that >>>>>> >> > you get socket operation on non-socket. I was hoping that >>>>>> someone else >>>>>> >> > could >>>>>> >> > help me find out why it is getting that error, and then may >>>>>> suggest what >>>>>> >> > I >>>>>> >> > change in libzmq. >>>>>> >> > >>>>>> >> > _______________________________________________ >>>>>> >> > zeromq-dev mailing list >>>>>> >> > [email protected] >>>>>> >> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>>> >> > >>>>>> >> _______________________________________________ >>>>>> >> zeromq-dev mailing list >>>>>> >> [email protected] >>>>>> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>>> > >>>>>> > >>>>>> > >>>>>> > _______________________________________________ >>>>>> > zeromq-dev mailing list >>>>>> > [email protected] >>>>>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>>> > >>>>>> _______________________________________________ >>>>>> zeromq-dev mailing list >>>>>> [email protected] >>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> zeromq-dev mailing list >>>>> [email protected] >>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> zeromq-dev mailing list >>>> [email protected] >>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>> >>>> >>> >>> _______________________________________________ >>> zeromq-dev mailing list >>> [email protected] >>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> >>> >> >> _______________________________________________ >> zeromq-dev mailing list >> [email protected] >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> >
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
