Re: [DynInst_API:] Telling DynInst a particular function is non-returning

[Thomas Dullien]

Hey there,

excellent, and thanks for the quick reply :-)

I will send a pull request for an easy cleanup (3) :-) and not
reformat until there's a good substantial change.

On (1), I have dug a bit further:

It appears that "Object" contains two relocation tables as members,
fbt_ and relocation_table_. Symtab seems to initialize
it's own member relocation_table_ by copying the contents of
Object.relocation_table_ in the one of the constructors; in
extract_info() it
seems to use Object.fbt_. The constructor that is hit in my case
(around line 1245 in 9.3.2) does not copy any entries. If I add
an extra loop to do it, at least the fbt_ in
SymtabCodeSource::init_linkage is no longer empty -- but the offsets
in the relocationEntry
are slightly off then, e.g. we'd need to use rel_addr() instead of
target_addr() as key?

I hope this helps somewhat,
Cheers,
Thomas

On Mon, Mar 19, 2018 at 6:56 PM, Bill Williams <b...@cs.wisc.edu> wrote:
> For 1) I'd have to do some code diving, but I can do a lightning round of 
> policy answers for 2-5:
>
> 2) There's not presently good documentation for this code, though I've got a 
> paper on the back burner that will address that somewhat. I can give you an 
> overview later when I'm not fighting all the fires.
> 3) Easy cleanups should always be fine.
> 4) We tend to prefer that reformatting come along with some substantial 
> changes (or at a minimum substantial review). Please treat four space indents 
> as standard if you do this. The worse the formatting of a file is the lower 
> the bar for fixing it.
> 5) Sure, lifting things to better/cleaner style is also always fine.
>
> --bw
> 
> From: Dyninst-api <dyninst-api-boun...@cs.wisc.edu> on behalf of Thomas 
> Dullien <thomasdull...@google.com>
> Sent: Monday, March 19, 2018 11:08 AM
> To: Xiaozhu Meng
> Cc: dyninst-api
> Subject: Re: [DynInst_API:] Telling DynInst a particular function is 
> non-returning
>
> Hey there,
>
> after some more digging, some more questions :-) -- some pertaining to
> my concrete question, some to the codebase
> in general.
>
> 1) I *think* I have identified the problem:
>
> Parser.C
> 1009: is_plt = HASHDEF(plt_entries,target);
> 1011: // CodeSource-defined tests
> 1012: is_nonret = obj().cs()->nonReturning(target);
>
> In my example run, plt_entries is empty when the code evaluates
> target=0x1030 (which is a jmp in the plt
> to __stack_chk_fail). Which part of the code would be normally
> responsible for initializing the plt entries? (see attached
> binary).
>
> This seems to happen because obj.cs()->linkage() seems to be empty
> during the construction of the Parser.
>
> The underlying culprit seems to be somewhere in SymtabCodeSource.C --
> normally, one would think that init_linkage
> would be responsible for filling the map. This function calls into
> Symtab::getFuncBindingTable, which ... strangely ...
> seems to only deal with a relocation table, not with a PLT ?
>
> So my question is: What part of the code is responsible for
> initializing the table with plt entries? :-)
>
> 2) Is there any documentation or rough overview about how the
> disassembler inside ParseAPI works? It seems
> the "workhorse" is parse_frames and parse_frame, but there is a fair
> bit of complexity in those functions, and not
> trivial to get a high-level overview :-)
>
> 3) Is it OK to submit pull requests with (semantics-preserving) code
> cleanups? There are a few places with
> pretty ugly hacks (for example the strcmp-statements around
> SymtabCodeSource.C below the "Achin added code" comment),
> and if nobody minds I'd clean them up :-)
>
> 4) Is it OK to submit pull requests with reformatting? The code seems
> to currently mix tabs and spaces in different files,
> and standardizing indentation on one of the two would make sense? The
> downside of such reformatting-commits is that
> they mess with git blame ...
>
> 5) Is it OK to submit pull requests that clean up iterator-based loops
> to for-in loops? E.g. is it OK to lift the code to use
> C++11 whenever it makes the code clearer / prettier?
>
> Cheers,
> Thomas
>
>
> On Fri, Mar 16, 2018 at 4:56 PM, Thomas Dullien
> <thomasdull...@google.com> wrote:
>> Hey there,
>>
>> update: It seems that the obj.cs()->linkage() at the beginning of
>> Parser::Parser returns an empty
>> plt ?
>>
>> Cheers,
>> Thomas
>>
>> On Fri, Mar 16, 2018 at 4:39 PM, Thomas Dullien <thomasdull...@google.com>
>> wrote:
>>>
>>> Hey there,
>>>
>>> ok, tracing this a bit further, I have some more questions :-)
>>>
>>>

Re: [DynInst_API:] Telling DynInst a particular function is non-returning

[Thomas Dullien]

Hey there,

update: It seems that the obj.cs()->linkage() at the beginning of
Parser::Parser returns an empty
plt ?

Cheers,
Thomas

On Fri, Mar 16, 2018 at 4:39 PM, Thomas Dullien <thomasdull...@google.com>
wrote:

> Hey there,
>
> ok, tracing this a bit further, I have some more questions :-)
>
> If I am not mistaken, the code around Parser.C, line 1013 is supposed to
> deal with calls to non-returning targets?
>
> is_nonret = obj().cs()->nonReturning(target);
> if (is_nonret) {
>  
>
> Now, in my example, for some reason the .plt.got stub function that jmps
> straight to cs:__stack_chk_fail is not a
> function:
>
> .plt.got:1030 __stack_chk_fail proc near  ; CODE
> XREF: postproc_version:loc_11D0↓p
> .plt.got:1030 ;
> postproc_configuration:loc_1202↓p ...
> .plt.got:1030 jmp cs:__stack_chk_fail_ptr
> .plt.got:1030 __stack_chk_fail endp
>
> So when nonReturning(0x1030) is called, no function gets returned in
> SymtabCodeSource::nonReturning while
> calling findFuncByEntryOffset, and then the code mistakenly adds the
> fallthrough edge.
>
> I am still finding my way through the codebase - but I presume there is
> some initial pass that is supposed to create
> functions from the stubs in the PLT and check them for non-returning
> status?
>
> Cheers,
> Thomas
>
>
>
> On Thu, Mar 15, 2018 at 6:33 PM, Thomas Dullien <thomasdull...@google.com>
> wrote:
>
>> Hey there,
>>
>> one quick question: After backporting the 3 commits, I tested with both
>> recursive-mode disassembly and non-recursive mode
>> disassembly, and neither of the two worked. My question is now: Is this
>> even supposed to work in non-recursive mode? E.g.
>> if we are not recursing into a called function, we can't tell whether it
>> returns or not, right?
>>
>> Cheers,
>> Thomas
>>
>> On Thu, Mar 15, 2018 at 2:23 PM, Thomas Dullien <thomasdull...@google.com
>> > wrote:
>>
>>> Hey there,
>>>
>>> ah, cool :-). I backported the 3 commits into the 9.3.2 codebase, but it
>>> does not seem to solve the issue just yet.
>>>
>>> I will debug a bit more and report back once I understand what is going
>>> on.
>>>
>>> Cheers,
>>> Thomas
>>>
>>> On Thu, Mar 15, 2018 at 2:09 PM, Xiaozhu Meng <xm...@cs.wisc.edu> wrote:
>>>
>>>> Ah, I have a few pending commits related to this issue. Please look at
>>>> my pull request at https://github.com/dyninst/dyninst/pull/437.
>>>>
>>>> Not all commits are relevant to you, but bbac557, 78b1bd1, da50a37 should
>>>> be relevant.
>>>>
>>>> For parsing_printf, it is "DYNINST_DEBUG_PARSING".
>>>>
>>>>
>>>> On Thu, Mar 15, 2018 at 8:02 AM, Thomas Dullien <
>>>> thomasdull...@google.com> wrote:
>>>>
>>>>> Hey there,
>>>>>
>>>>> ok, rebuilding DynInst 9.3.2 now to investigate why adding the string
>>>>> does not seem to have any effect.
>>>>> I looked at the source code, and I *think* the function is already in
>>>>> the list of non-returning functions.
>>>>>
>>>>> Checking what is going on. Example setup:
>>>>>
>>>>> .plt.got:00053CF8 __stack_chk_fail proc near  ;
>>>>> CODE XREF: init:loc_54673↓p
>>>>> .plt.got:00053CF8 ;
>>>>> uninit:loc_54705↓p ...
>>>>> .plt.got:00053CF8 jmp
>>>>>  cs:__stack_chk_fail_ptr
>>>>> .plt.got:00053CF8 __stack_chk_fail endp
>>>>>
>>>>> .text:000A4290 var_8   = qword ptr -8
>>>>> .text:000A4290
>>>>> .text:000A4290 graph = rdi ;
>>>>> AVFilterGraph_0 *
>>>>> .text:000A4290 flags = rsi ;
>>>>> unsigned int
>>>>> .text:000A4290 ; __unwind {
>>>>> .text:000A4290 pushrax
>>>>> .text:000A4291 mov rax, fs:28h
>>>>> .text:000A429A mov [rsp+8+var_8], rax
>>>>> .text:000A429E mov [graph+5Ch], esi
>>>>> .text:000A42A1  

Re: [DynInst_API:] Telling DynInst a particular function is non-returning

[Thomas Dullien]

Hey there,

ok, tracing this a bit further, I have some more questions :-)

If I am not mistaken, the code around Parser.C, line 1013 is supposed to
deal with calls to non-returning targets?

is_nonret = obj().cs()->nonReturning(target);
if (is_nonret) {
 

Now, in my example, for some reason the .plt.got stub function that jmps
straight to cs:__stack_chk_fail is not a
function:

.plt.got:1030 __stack_chk_fail proc near  ; CODE
XREF: postproc_version:loc_11D0↓p
.plt.got:1030 ;
postproc_configuration:loc_1202↓p ...
.plt.got:1030 jmp cs:__stack_chk_fail_ptr
.plt.got:1030 __stack_chk_fail endp

So when nonReturning(0x1030) is called, no function gets returned in
SymtabCodeSource::nonReturning while
calling findFuncByEntryOffset, and then the code mistakenly adds the
fallthrough edge.

I am still finding my way through the codebase - but I presume there is
some initial pass that is supposed to create
functions from the stubs in the PLT and check them for non-returning status?

Cheers,
Thomas



On Thu, Mar 15, 2018 at 6:33 PM, Thomas Dullien <thomasdull...@google.com>
wrote:

> Hey there,
>
> one quick question: After backporting the 3 commits, I tested with both
> recursive-mode disassembly and non-recursive mode
> disassembly, and neither of the two worked. My question is now: Is this
> even supposed to work in non-recursive mode? E.g.
> if we are not recursing into a called function, we can't tell whether it
> returns or not, right?
>
> Cheers,
> Thomas
>
> On Thu, Mar 15, 2018 at 2:23 PM, Thomas Dullien <thomasdull...@google.com>
> wrote:
>
>> Hey there,
>>
>> ah, cool :-). I backported the 3 commits into the 9.3.2 codebase, but it
>> does not seem to solve the issue just yet.
>>
>> I will debug a bit more and report back once I understand what is going
>> on.
>>
>> Cheers,
>> Thomas
>>
>> On Thu, Mar 15, 2018 at 2:09 PM, Xiaozhu Meng <xm...@cs.wisc.edu> wrote:
>>
>>> Ah, I have a few pending commits related to this issue. Please look at
>>> my pull request at https://github.com/dyninst/dyninst/pull/437.
>>>
>>> Not all commits are relevant to you, but bbac557, 78b1bd1, da50a37 should
>>> be relevant.
>>>
>>> For parsing_printf, it is "DYNINST_DEBUG_PARSING".
>>>
>>>
>>> On Thu, Mar 15, 2018 at 8:02 AM, Thomas Dullien <
>>> thomasdull...@google.com> wrote:
>>>
>>>> Hey there,
>>>>
>>>> ok, rebuilding DynInst 9.3.2 now to investigate why adding the string
>>>> does not seem to have any effect.
>>>> I looked at the source code, and I *think* the function is already in
>>>> the list of non-returning functions.
>>>>
>>>> Checking what is going on. Example setup:
>>>>
>>>> .plt.got:00053CF8 __stack_chk_fail proc near  ;
>>>> CODE XREF: init:loc_54673↓p
>>>> .plt.got:00053CF8 ;
>>>> uninit:loc_54705↓p ...
>>>> .plt.got:00053CF8 jmp
>>>>  cs:__stack_chk_fail_ptr
>>>> .plt.got:00053CF8 __stack_chk_fail endp
>>>>
>>>> .text:000A4290 var_8   = qword ptr -8
>>>> .text:000A4290
>>>> .text:000A4290 graph = rdi ;
>>>> AVFilterGraph_0 *
>>>> .text:000A4290 flags = rsi ;
>>>> unsigned int
>>>> .text:000A4290 ; __unwind {
>>>> .text:000A4290 pushrax
>>>> .text:000A4291 mov rax, fs:28h
>>>> .text:000A429A mov [rsp+8+var_8], rax
>>>> .text:000A429E mov [graph+5Ch], esi
>>>> .text:000A42A1 mov rax, fs:28h
>>>> .text:000A42AA cmp rax, [rsp+8+var_8]
>>>> .text:000A42AE jnz short loc_A42B2
>>>> .text:000A42B0 pop rax
>>>> .text:000A42B1 retn
>>>> .text:000A42B2 ; --
>>>> -----
>>>> .text:000A42B2
>>>> .text:000A42B2 loc_A42B2:  ; CODE
>>>> XREF: avfilter_graph_set_auto_convert+1E↑j
>>>> .text:000A42B2   

Re: [DynInst_API:] Telling DynInst a particular function is non-returning

[Thomas Dullien]

Hey there,

one quick question: After backporting the 3 commits, I tested with both
recursive-mode disassembly and non-recursive mode
disassembly, and neither of the two worked. My question is now: Is this
even supposed to work in non-recursive mode? E.g.
if we are not recursing into a called function, we can't tell whether it
returns or not, right?

Cheers,
Thomas

On Thu, Mar 15, 2018 at 2:23 PM, Thomas Dullien <thomasdull...@google.com>
wrote:

> Hey there,
>
> ah, cool :-). I backported the 3 commits into the 9.3.2 codebase, but it
> does not seem to solve the issue just yet.
>
> I will debug a bit more and report back once I understand what is going on.
>
> Cheers,
> Thomas
>
> On Thu, Mar 15, 2018 at 2:09 PM, Xiaozhu Meng <xm...@cs.wisc.edu> wrote:
>
>> Ah, I have a few pending commits related to this issue. Please look at my
>> pull request at https://github.com/dyninst/dyninst/pull/437.
>>
>> Not all commits are relevant to you, but bbac557, 78b1bd1, da50a37 should
>> be relevant.
>>
>> For parsing_printf, it is "DYNINST_DEBUG_PARSING".
>>
>>
>> On Thu, Mar 15, 2018 at 8:02 AM, Thomas Dullien <thomasdull...@google.com
>> > wrote:
>>
>>> Hey there,
>>>
>>> ok, rebuilding DynInst 9.3.2 now to investigate why adding the string
>>> does not seem to have any effect.
>>> I looked at the source code, and I *think* the function is already in
>>> the list of non-returning functions.
>>>
>>> Checking what is going on. Example setup:
>>>
>>> .plt.got:00053CF8 __stack_chk_fail proc near  ; CODE
>>> XREF: init:loc_54673↓p
>>> .plt.got:00053CF8 ;
>>> uninit:loc_54705↓p ...
>>> .plt.got:00053CF8 jmp cs:__stack_chk_fail_ptr
>>> .plt.got:00053CF8 __stack_chk_fail endp
>>>
>>> .text:000A4290 var_8   = qword ptr -8
>>> .text:000A4290
>>> .text:000A4290 graph = rdi ;
>>> AVFilterGraph_0 *
>>> .text:000A4290 flags = rsi ;
>>> unsigned int
>>> .text:000A4290 ; __unwind {
>>> .text:000A4290 pushrax
>>> .text:000A4291 mov rax, fs:28h
>>> .text:000A429A mov [rsp+8+var_8], rax
>>> .text:000A429E mov [graph+5Ch], esi
>>> .text:000A42A1 mov rax, fs:28h
>>> .text:000A42AA cmp rax, [rsp+8+var_8]
>>> .text:000A42AE jnz short loc_A42B2
>>> .text:000A42B0 pop rax
>>> .text:000A42B1 retn
>>> .text:000A42B2 ; --
>>> -
>>> .text:000A42B2
>>> .text:000A42B2 loc_A42B2:  ; CODE
>>> XREF: avfilter_graph_set_auto_convert+1E↑j
>>> .text:000A42B2 call__stack_chk_fail
>>> .text:000A42B2 ; } // starts at A4290
>>> .text:000A42B2 avfilter_graph_set_auto_convert endp
>>>
>>> DynInst for some reason does not interpret the tailing call as
>>> non-return. Digging to see what is going on,
>>> will update here as I learn more :)
>>>
>>> Cheers,
>>> Thomas
>>>
>>>
>>> On Thu, Mar 15, 2018 at 1:58 PM, Xiaozhu Meng <mxz...@gmail.com> wrote:
>>>
>>>> Hi Thomas,
>>>>
>>>> On Thu, Mar 15, 2018 at 4:06 AM, Thomas Dullien <
>>>> thomasdull...@google.com> wrote:
>>>>
>>>>> Hey there,
>>>>>
>>>>> ok, I have looked at a few options on how to best tackle this, and
>>>>> would love to solicit advice :-)
>>>>>
>>>>> - I tried SymtabCodeSource::addNonReturning("__stack_chk_fail"); this
>>>>> did not seem to have an effect.
>>>>>
>>>>
>>>> If you call SymtabCodeSource::addNonReturning("__stack_chk_fail")
>>>> before calling CodeObject::parse(), this should work.
>>>>
>>>>
>>>>> - Looked at set_retstatus -- but that implies that the code is already
>>>>> parsed?
>>>>>
>>>>
>>>> You are right that after CodeObject::pa

Re: [DynInst_API:] Telling DynInst a particular function is non-returning

[Thomas Dullien]

Hey there,

ah, cool :-). I backported the 3 commits into the 9.3.2 codebase, but it
does not seem to solve the issue just yet.

I will debug a bit more and report back once I understand what is going on.

Cheers,
Thomas

On Thu, Mar 15, 2018 at 2:09 PM, Xiaozhu Meng <xm...@cs.wisc.edu> wrote:

> Ah, I have a few pending commits related to this issue. Please look at my
> pull request at https://github.com/dyninst/dyninst/pull/437.
>
> Not all commits are relevant to you, but bbac557, 78b1bd1, da50a37 should
> be relevant.
>
> For parsing_printf, it is "DYNINST_DEBUG_PARSING".
>
>
> On Thu, Mar 15, 2018 at 8:02 AM, Thomas Dullien <thomasdull...@google.com>
> wrote:
>
>> Hey there,
>>
>> ok, rebuilding DynInst 9.3.2 now to investigate why adding the string
>> does not seem to have any effect.
>> I looked at the source code, and I *think* the function is already in the
>> list of non-returning functions.
>>
>> Checking what is going on. Example setup:
>>
>> .plt.got:00053CF8 __stack_chk_fail proc near  ; CODE
>> XREF: init:loc_54673↓p
>> .plt.got:00053CF8 ;
>> uninit:loc_54705↓p ...
>> .plt.got:00053CF8 jmp cs:__stack_chk_fail_ptr
>> .plt.got:00053CF8 __stack_chk_fail endp
>>
>> .text:000A4290 var_8   = qword ptr -8
>> .text:000A4290
>> .text:000A4290 graph = rdi ;
>> AVFilterGraph_0 *
>> .text:000A4290 flags = rsi ; unsigned
>> int
>> .text:000A4290 ; __unwind {
>> .text:000A4290 pushrax
>> .text:000A4291 mov rax, fs:28h
>> .text:000A429A mov [rsp+8+var_8], rax
>> .text:000A429E mov [graph+5Ch], esi
>> .text:000A42A1 mov rax, fs:28h
>> .text:000A42AA cmp rax, [rsp+8+var_8]
>> .text:000A42AE jnz short loc_A42B2
>> .text:000A42B0 pop rax
>> .text:000A42B1 retn
>> .text:000A42B2 ; --
>> -
>> .text:000A42B2
>> .text:000A42B2 loc_A42B2:  ; CODE
>> XREF: avfilter_graph_set_auto_convert+1E↑j
>> .text:000A42B2 call__stack_chk_fail
>> .text:000A42B2 ; } // starts at A4290
>> .text:000A42B2 avfilter_graph_set_auto_convert endp
>>
>> DynInst for some reason does not interpret the tailing call as
>> non-return. Digging to see what is going on,
>> will update here as I learn more :)
>>
>> Cheers,
>> Thomas
>>
>>
>> On Thu, Mar 15, 2018 at 1:58 PM, Xiaozhu Meng <mxz...@gmail.com> wrote:
>>
>>> Hi Thomas,
>>>
>>> On Thu, Mar 15, 2018 at 4:06 AM, Thomas Dullien <
>>> thomasdull...@google.com> wrote:
>>>
>>>> Hey there,
>>>>
>>>> ok, I have looked at a few options on how to best tackle this, and
>>>> would love to solicit advice :-)
>>>>
>>>> - I tried SymtabCodeSource::addNonReturning("__stack_chk_fail"); this
>>>> did not seem to have an effect.
>>>>
>>>
>>> If you call SymtabCodeSource::addNonReturning("__stack_chk_fail")
>>> before calling CodeObject::parse(), this should work.
>>>
>>>
>>>> - Looked at set_retstatus -- but that implies that the code is already
>>>> parsed?
>>>>
>>>
>>> You are right that after CodeObject::parse() has finished, calling
>>> set_retstatus will only change the flag of return status for this function,
>>> will not re-parse the binary. So, we can focus on why SymtabCodeSource::
>>> addNonReturning("__stack_chk_fail") does not work.
>>>
>>>
>>>
>>>
>>>>
>>>> Cheers,
>>>> Thomas
>>>>
>>>>
>>>> On Thu, Mar 15, 2018 at 9:28 AM, Thomas Dullien <
>>>> thomasdull...@google.com> wrote:
>>>>
>>>>> Ah. No. I just fund set_retstatus :-) -- please ignore my question :-)
>>>>>
>>>>> On Thu, Mar 15, 2018 at 9:26 AM, Thomas Dullien <
>>>>> thomasdull...@google.com> wrote:
>>>>>
>>>>>> Hey there,
>&

Re: [DynInst_API:] Telling DynInst a particular function is non-returning

[Thomas Dullien]

Hey there,

ok, rebuilding DynInst 9.3.2 now to investigate why adding the string does
not seem to have any effect.
I looked at the source code, and I *think* the function is already in the
list of non-returning functions.

Checking what is going on. Example setup:

.plt.got:00053CF8 __stack_chk_fail proc near  ; CODE
XREF: init:loc_54673↓p
.plt.got:00053CF8 ;
uninit:loc_54705↓p ...
.plt.got:00053CF8 jmp cs:__stack_chk_fail_ptr
.plt.got:00053CF8 __stack_chk_fail endp

.text:000A4290 var_8   = qword ptr -8
.text:000A4290
.text:000A4290 graph = rdi ;
AVFilterGraph_0 *
.text:000A4290 flags = rsi ; unsigned
int
.text:000A4290 ; __unwind {
.text:000A4290 pushrax
.text:000A4291 mov rax, fs:28h
.text:000A429A mov [rsp+8+var_8], rax
.text:000A429E mov [graph+5Ch], esi
.text:000A42A1 mov rax, fs:28h
.text:000A42AA cmp rax, [rsp+8+var_8]
.text:000A42AE jnz short loc_A42B2
.text:000A42B0 pop rax
.text:000A42B1 retn
.text:000A42B2 ;
---
.text:000A42B2
.text:000A42B2 loc_A42B2:  ; CODE XREF:
avfilter_graph_set_auto_convert+1E↑j
.text:000A42B2 call__stack_chk_fail
.text:000A42B2 ; } // starts at A4290
.text:000A42B2 avfilter_graph_set_auto_convert endp

DynInst for some reason does not interpret the tailing call as non-return.
Digging to see what is going on,
will update here as I learn more :)

Cheers,
Thomas


On Thu, Mar 15, 2018 at 1:58 PM, Xiaozhu Meng <mxz...@gmail.com> wrote:

> Hi Thomas,
>
> On Thu, Mar 15, 2018 at 4:06 AM, Thomas Dullien <thomasdull...@google.com>
> wrote:
>
>> Hey there,
>>
>> ok, I have looked at a few options on how to best tackle this, and would
>> love to solicit advice :-)
>>
>> - I tried SymtabCodeSource::addNonReturning("__stack_chk_fail"); this
>> did not seem to have an effect.
>>
>
> If you call SymtabCodeSource::addNonReturning("__stack_chk_fail") before
> calling CodeObject::parse(), this should work.
>
>
>> - Looked at set_retstatus -- but that implies that the code is already
>> parsed?
>>
>
> You are right that after CodeObject::parse() has finished, calling
> set_retstatus will only change the flag of return status for this function,
> will not re-parse the binary. So, we can focus on why SymtabCodeSource::
> addNonReturning("__stack_chk_fail") does not work.
>
>
>
>
>>
>> Cheers,
>> Thomas
>>
>>
>> On Thu, Mar 15, 2018 at 9:28 AM, Thomas Dullien <thomasdull...@google.com
>> > wrote:
>>
>>> Ah. No. I just fund set_retstatus :-) -- please ignore my question :-)
>>>
>>> On Thu, Mar 15, 2018 at 9:26 AM, Thomas Dullien <
>>> thomasdull...@google.com> wrote:
>>>
>>>> Hey there,
>>>>
>>>> after having my coffee, I realized: The proper way to do this is to
>>>> derive from CodeSource
>>>> and overload the nonReturning functions, I guess? :)
>>>>
>>>> Cheers,
>>>> Thomas
>>>>
>>>> On Thu, Mar 15, 2018 at 9:14 AM, Thomas Dullien <
>>>> thomasdull...@google.com> wrote:
>>>>
>>>>> Hey there,
>>>>>
>>>>> I am running into troubles with disassembling executables generated by
>>>>> clang.3.8.1-24, for x64, with optimization set to size-optimize and
>>>>> stack cookies
>>>>> enabled.
>>>>>
>>>>> The trouble is that any function with an enabled stack cookie will end
>>>>> with a sequence
>>>>> of:
>>>>>
>>>>>   Epilogue to check stack cookie
>>>>>   jnz .fail
>>>>>   Rest of epilogue.
>>>>>   retn
>>>>> .fail:
>>>>>   call __stack_checkfail // Does not return
>>>>>
>>>>> This leads to DynInst lumping all consecutive functions that use stack
>>>>> cookies
>>>>> into one huge CFG.
>>>>>
>>>>> Is there a way to tell DynInst that a particular function is not
>>>>> returning? I found
>>>>> that the parseAPI allows me to query if a function returns, but I did
>>>>> not find anything
>>>>> to "override" this behavior?
>>>>>
>>>>> Cheers,
>>>>> Thomas
>>>>>
>>>>
>>>>
>>>
>>
>> ___
>> Dyninst-api mailing list
>> Dyninst-api@cs.wisc.edu
>> https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api
>>
>
>
___
Dyninst-api mailing list
Dyninst-api@cs.wisc.edu
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api

Re: [DynInst_API:] Telling DynInst a particular function is non-returning

[Thomas Dullien]

Hey there,

after having my coffee, I realized: The proper way to do this is to derive
from CodeSource
and overload the nonReturning functions, I guess? :)

Cheers,
Thomas

On Thu, Mar 15, 2018 at 9:14 AM, Thomas Dullien <thomasdull...@google.com>
wrote:

> Hey there,
>
> I am running into troubles with disassembling executables generated by
> clang.3.8.1-24, for x64, with optimization set to size-optimize and stack
> cookies
> enabled.
>
> The trouble is that any function with an enabled stack cookie will end
> with a sequence
> of:
>
>   Epilogue to check stack cookie
>   jnz .fail
>   Rest of epilogue.
>   retn
> .fail:
>   call __stack_checkfail // Does not return
>
> This leads to DynInst lumping all consecutive functions that use stack
> cookies
> into one huge CFG.
>
> Is there a way to tell DynInst that a particular function is not
> returning? I found
> that the parseAPI allows me to query if a function returns, but I did not
> find anything
> to "override" this behavior?
>
> Cheers,
> Thomas
>
___
Dyninst-api mailing list
Dyninst-api@cs.wisc.edu
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api

[DynInst_API:] Telling DynInst a particular function is non-returning

[Thomas Dullien]

Hey there,

I am running into troubles with disassembling executables generated by
clang.3.8.1-24, for x64, with optimization set to size-optimize and stack
cookies
enabled.

The trouble is that any function with an enabled stack cookie will end with
a sequence
of:

  Epilogue to check stack cookie
  jnz .fail
  Rest of epilogue.
  retn
.fail:
  call __stack_checkfail // Does not return

This leads to DynInst lumping all consecutive functions that use stack
cookies
into one huge CFG.

Is there a way to tell DynInst that a particular function is not returning?
I found
that the parseAPI allows me to query if a function returns, but I did not
find anything
to "override" this behavior?

Cheers,
Thomas
___
Dyninst-api mailing list
Dyninst-api@cs.wisc.edu
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api

Re: [DynInst_API:] segfault when trying to parse an ELF, dereferencing [0] in vector of size 0

[Thomas Dullien]

Hey there,

a quick update: The issue seems to be that the ELF file has legitimate
shdrnum=0 in the header,
which some of the other code is not prepared for.

Cheers,
Thomas

On Thu, Jan 11, 2018 at 11:36 AM, Thomas Dullien <thomasdull...@google.com>
wrote:

> Hey there,
>
> I am trying to run a particular ELF file through Dyninst, but I get a
> segfault
> in Elf_X.C:
>
> if (!shdrs[i]._elf) {
>
> The trouble is that shdrs is an empty vector, and i is 0, so we can't
> dereference
> at index 0.
>
> #0  Dyninst::Elf_X::get_shdr (this=this@entry=0x557c15ecf0a0, i=0) at
> /home/thomasdullien/Desktop/sources/dyninst/dyninst/elf/src/Elf_X.C:473
> #1  0x7fd72b83f43a in Dyninst::Elf_X::findDebugFile
> (this=0x557c15ecf0a0, origfilename="/home/thomasdullien/Downloads/foo.elf",
> output_name="", output_buffer=@0x7fff5398c320: 0x557c15ecf470
> "/home/thomasdullien/Downloads/foo.elf", output_buffer_size=@0x7fff5398c
> 328: 93991432156048) at /home/thomasdullien/Desktop/
> sources/dyninst/dyninst/elf/src/Elf_X.C:1659
> #2  0x7fd72ba5b124 in Dyninst::Dwarf::DwarfHandle::locate_dbg_file
> (this=this@entry=0x557c15ecf3d0) at /home/thomasdullien/Desktop/
> sources/dyninst/dyninst/dwarf/src/dwarfHandle.C:80
> #3  0x7fd72ba5b29f in Dyninst::Dwarf::DwarfHandle::DwarfHandle
> (this=0x557c15ecf3d0, filename_=..., file_=,
> err_func_=) at /home/thomasdullien/Desktop/
> sources/dyninst/dyninst/dwarf/src/dwarfHandle.C:73
> #4  0x7fd72ba5b9ff in Dyninst::Dwarf::DwarfHandle::createDwarfHandle
> (filename_="/home/thomasdullien/Downloads/foo.elf", 
> file_=file_@entry=0x557c15ecf0a0,
> err_func_=err_func_@entry=0x7fd72ba5b010  defaultDwarfError(Dwarf_Error_s*, void*
> )>) at /home/thomasdullien/Desktop/sources/dyninst/dyninst/dwarf/
> src/dwarfHandle.C:249
>
> Shall I try to dig further, or should this be sufficient to find the
> problem?
>
> Cheers & keep up the good work :-)
> Thomas
>
>
___
Dyninst-api mailing list
Dyninst-api@cs.wisc.edu
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api

[DynInst_API:] segfault when trying to parse an ELF, dereferencing [0] in vector of size 0

[Thomas Dullien]

Hey there,

I am trying to run a particular ELF file through Dyninst, but I get a
segfault
in Elf_X.C:

if (!shdrs[i]._elf) {

The trouble is that shdrs is an empty vector, and i is 0, so we can't
dereference
at index 0.

#0  Dyninst::Elf_X::get_shdr (this=this@entry=0x557c15ecf0a0, i=0) at
/home/thomasdullien/Desktop/sources/dyninst/dyninst/elf/src/Elf_X.C:473
#1  0x7fd72b83f43a in Dyninst::Elf_X::findDebugFile
(this=0x557c15ecf0a0, origfilename="/home/thomasdullien/Downloads/foo.elf",
output_name="", output_buffer=@0x7fff5398c320: 0x557c15ecf470
"/home/thomasdullien/Downloads/foo.elf", output_buffer_size=@0x7fff5398c
328: 93991432156048) at
/home/thomasdullien/Desktop/sources/dyninst/dyninst/elf/src/Elf_X.C:1659
#2  0x7fd72ba5b124 in Dyninst::Dwarf::DwarfHandle::locate_dbg_file
(this=this@entry=0x557c15ecf3d0) at
/home/thomasdullien/Desktop/sources/dyninst/dyninst/dwarf/src/dwarfHandle.C:80
#3  0x7fd72ba5b29f in Dyninst::Dwarf::DwarfHandle::DwarfHandle
(this=0x557c15ecf3d0, filename_=..., file_=,
err_func_=) at
/home/thomasdullien/Desktop/sources/dyninst/dyninst/dwarf/src/dwarfHandle.C:73
#4  0x7fd72ba5b9ff in Dyninst::Dwarf::DwarfHandle::createDwarfHandle
(filename_="/home/thomasdullien/Downloads/foo.elf",
file_=file_@entry=0x557c15ecf0a0,
err_func_=err_func_@entry=0x7fd72ba5b010
) at
/home/thomasdullien/Desktop/sources/dyninst/dyninst/dwarf/src/dwarfHandle.C:249

Shall I try to dig further, or should this be sufficient to find the
problem?

Cheers & keep up the good work :-)
Thomas
___
Dyninst-api mailing list
Dyninst-api@cs.wisc.edu
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api

Re: [DynInst_API:] Problems with cotire

[Thomas Dullien]

Hey there,

thanks a ton :-). make clean_cotire did not work, but I am recompiling
everything from scratch now (with ENABLE_COTIRE=false), and
that seems to work :-)

Cheers,
Thomas

On Mon, Dec 11, 2017 at 3:06 PM, Bill Williams <b...@cs.wisc.edu> wrote:

> There's an ENABLE_COTIRE or USE_COTIRE in the top-level CMakeLists that
> can easily be promoted to a visible cache variable if you like.
>
> You can also try make clean_cotire IIRC (check your makefile for the
> target name) if this is a system upgrade in place and you've not cleaned
> out PCHes from pre-upgrade. That should get things moving; I presume you've
> upgraded well past gcc 4.8 now.
> 
> From: Dyninst-api <dyninst-api-boun...@cs.wisc.edu> on behalf of Thomas
> Dullien <thomasdull...@google.com>
> Sent: Monday, December 11, 2017 4:46 PM
> To: dyninst-api
> Subject: [DynInst_API:] Problems with cotire
>
> Hey there,
>
> I upgraded my operating system yesterday, and am running into
> problems building dyninst under buster/sid today. I am getting the
> following errors and have been trying all sorts of things to fix the
> issues -- does anyone have any advice?
>
> [  5%] Built target DyninstRT
> [  5%] Generating CXX unity source common/cotire/common_CXX_unity.cxx
> [  5%] Generating CXX prefix source common/cotire/common_CXX_prefix.cxx
> [  5%] Generating CXX prefix header common/cotire/common_CXX_prefix.hxx
> make[2]: *** No rule to make target '/usr/include/c++/4.8/algorithm',
> needed by 'common/cotire/common_CXX_prefix.hxx.gch'.  Stop.
> CMakeFiles/Makefile2:566: recipe for target 'common/CMakeFiles/common.dir/all'
> failed
> make[1]: *** [common/CMakeFiles/common.dir/all] Error 2
> Makefile:127: recipe for target 'all' failed
> make: *** [all] Error 2
>
> Is there a way to disable precompiled headers generally for the
> compilation process?
> I am fine with longer compiles :-)
>
> Cheers,
> Thomas
>
___
Dyninst-api mailing list
Dyninst-api@cs.wisc.edu
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api

[DynInst_API:] Problems with cotire

[Thomas Dullien]

Hey there,

I upgraded my operating system yesterday, and am running into
problems building dyninst under buster/sid today. I am getting the
following errors and have been trying all sorts of things to fix the
issues -- does anyone have any advice?

[  5%] Built target DyninstRT
[  5%] Generating CXX unity source common/cotire/common_CXX_unity.cxx
[  5%] Generating CXX prefix source common/cotire/common_CXX_prefix.cxx
[  5%] Generating CXX prefix header common/cotire/common_CXX_prefix.hxx
make[2]: *** No rule to make target '/usr/include/c++/4.8/algorithm',
needed by 'common/cotire/common_CXX_prefix.hxx.gch'.  Stop.
CMakeFiles/Makefile2:566: recipe for target
'common/CMakeFiles/common.dir/all' failed
make[1]: *** [common/CMakeFiles/common.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2

Is there a way to disable precompiled headers generally for the compilation
process?
I am fine with longer compiles :-)

Cheers,
Thomas
___
Dyninst-api mailing list
Dyninst-api@cs.wisc.edu
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api

Re: [DynInst_API:] Control Flow Graph example with parseAPI

[Thomas Dullien]

Hey there,

not sure if this helps, but there is a fair bit of code that uses the
ParseAPI and CFGs here:
https://github.com/google/functionsimsearch

In particular
https://github.com/google/functionsimsearch/blob/master/dotgraphs.cpp dumps
out
CFGs from an executable (ELF or PE) to .dot format for easy plotting.


Hope this helps,
Cheers,
Thomas

On Mon, Dec 4, 2017 at 12:16 AM, Buddhika Chamith Kahawitage Don <
budka...@umail.iu.edu> wrote:

> Sorry for the late response on this. Got side tracked with other pursuits.
> Anyways it would be just easier if these examples were in the form of
> executable tests/ examples within the Dyninst repository. Didn't see
> something like that though. Did I miss it? (I laboriously copied stuff from
> the pdf to assemble the example source. There is a chance I may have missed
> some stuff during that process)
>
> Thanks
> Buddhika
>
> On Mon, Nov 13, 2017 at 9:42 AM, Xiaozhu Meng  wrote:
>
>> Hi Buddhika,
>>
>> I have not encountered such issue before. If you still have this problem,
>> you can send me you source file and ELF binary, and I will try to reproduce
>> your issue and see what is happening.
>>
>> Thanks,
>>
>> --Xiaozhu
>>
>> On Fri, Nov 10, 2017 at 7:08 PM, Buddhika Chamith Kahawitage Don <
>> budka...@umail.iu.edu> wrote:
>>
>>> Hi All,
>>>
>>> I was trying to run control flow graph example (3.2) in the parseAPI
>>> documentation. When I run on some ELF binary  I am getting lot of non ascii
>>> (binary) outputs instead of the intended output. Anyone come across this
>>> before?
>>>
>>> Thanks
>>> Buddhika
>>>
>>> ___
>>> Dyninst-api mailing list
>>> Dyninst-api@cs.wisc.edu
>>> https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api
>>>
>>
>>
>
> ___
> Dyninst-api mailing list
> Dyninst-api@cs.wisc.edu
> https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api
>
___
Dyninst-api mailing list
Dyninst-api@cs.wisc.edu
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api

Re: [DynInst_API:] Question on potential memory leaks

[Thomas Dullien]

Hey there,

that is an excellent question :-). As far as I know, yes, but I don't
*know* - and the fact that the same code
compiled using a different GCC version does not show the leaks gives
further pause.

I will investigate further -- at the moment, the quantities reported are
very non-threatening, (850k bytes
over a few dozen full disassemblies), so it is not an urgent question
anyhow :-)

Cheers,
Thomas

On Wed, Sep 27, 2017 at 4:44 PM, Xiaozhu Meng <mxz...@gmail.com> wrote:

> Hi Thomas,
>
> I had some memory issues related to ROSE in Dyninst, but they were not
> about memory leaks. My experiences are that ROSE internally has their own
> memory management system, including their customized small object
> allocation, deallocation, and their own implementation for shared pointers.
> ROSE typically free objects created by ROSE at the very end of the program,
> specifically, after returning from main and in global destruction time.
>
> I have little experience with LeakSanitizer, but my question is does
> LeakSanitizer consider code after returning from Main?
>
> Thanks,
>
> --Xiaozhu
>
> On Wed, Sep 27, 2017 at 9:03 AM, Thomas Dullien <thomasdull...@google.com>
> wrote:
>
>> Hey all,
>>
>> in the following two example leaks (there are lots in my output, but they
>> are all of one of these two types):
>>
>> Indirect leak of 8 byte(s) in 1 object(s) allocated from:
>> #0 0x7fb7aa213bf0 in operator new(unsigned long)
>> (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc2bf0)
>> #1 0x7fb7a9cc1329 in Dyninst::DataflowAPI::Expressi
>> onConversionVisitor::visit(Dyninst::InstructionAPI::Dereference*)
>> /home/thomasdullien/dyninst-9.3.2/dataflowAPI/src/Expression
>> ConversionVisitor.C:217
>> #2 0x7fb7a9cd6a6a in Dyninst::DataflowAPI::RoseInsn
>> Factory::convertOperand(boost::shared_ptr,
>> long, unsigned long) /home/thomasdullien/dyninst-9.
>> 3.2/dataflowAPI/src/RoseInsnFactory.C:105
>> #3 0x608000104e1f  ()
>>
>> Indirect leak of 9 byte(s) in 1 object(s) allocated from:
>> #0 0x7fb7aa213bf0 in operator new(unsigned long)
>> (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc2bf0)
>> #1 0x7fb7a9cd3307 in __gnu_cxx::new_allocator> char>::allocate(unsigned long, void const*) /usr/include/c++/6/ext/new_all
>> ocator.h:104
>> #2 0x7fb7a9cd3307 in std::allocator_traits<std::allocator> char> >::allocate(std::allocator&, unsigned long)
>> /usr/include/c++/6/bits/alloc_traits.h:436
>> #3 0x7fb7a9cd3307 in std::_Vector_base> std::allocator >::_M_allocate(unsigned long)
>> /usr/include/c++/6/bits/stl_vector.h:170
>> #4 0x7fb7a9cd3307 in unsigned char* std::vector> std::allocator >::_M_allocate_and_copy<__gnu_
>> cxx::__normal_iterator> std::allocator > > >(unsigned l
>> ong, __gnu_cxx::__normal_iterator> std::vector > >,
>> __gnu_cxx::__normal_iterator> char, std::allocator > >) /usr/include/c
>> ++/6/bits/stl_vector.h:1222
>> #5 0x7fb7a9cd3307 in std::vector> std::allocator >::operator=(std::vector> std::allocator > const&) /usr/include/c++/6/bits/vector
>> .tcc:195
>> #6 0x7fb7a9cd3307 in SgAsmInstruction::set_raw_bytes(std::vector> char, std::allocator >) /home/thomasdullien/dyninst-9.
>> 3.2/dataflowAPI/src/RoseImpl.C:1363
>> #7 0x602d4478  ()
>>
>> Do these seem like real leaks? (This is not from trunk, but the 9.3
>> release).
>>
>> Cheers,
>> Thomas
>>
>> On Wed, Sep 27, 2017 at 3:59 PM, Thomas Dullien <thomasdull...@google.com
>> > wrote:
>>
>>> Hey there,
>>>
>>> while tracing down some undesired behavior in Functionsimsearch, I
>>> compiled and
>>> ran everything under gcc6 and -fsanitize=address, which includes
>>> LeakSanitizer.
>>>
>>> I get a fairly large-ish number of reported leaks from Rose:
>>>
>>>
>>
>> ___
>> Dyninst-api mailing list
>> Dyninst-api@cs.wisc.edu
>> https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api
>>
>>
>
___
Dyninst-api mailing list
Dyninst-api@cs.wisc.edu
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api

Re: [DynInst_API:] Question on potential memory leaks

[Thomas Dullien]

Hey all,

in the following two example leaks (there are lots in my output, but they
are all of one of these two types):

Indirect leak of 8 byte(s) in 1 object(s) allocated from:
#0 0x7fb7aa213bf0 in operator new(unsigned long)
(/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc2bf0)
#1 0x7fb7a9cc1329 in
Dyninst::DataflowAPI::ExpressionConversionVisitor::visit(Dyninst::InstructionAPI::Dereference*)
/home/thomasdullien/dyninst-9.3.2/dataflowAPI/src/ExpressionConversionVisitor.C:217
#2 0x7fb7a9cd6a6a in
Dyninst::DataflowAPI::RoseInsnFactory::convertOperand(boost::shared_ptr,
long, unsigned long)
/home/thomasdullien/dyninst-9.3.2/dataflowAPI/src/RoseInsnFactory.C:105
#3 0x608000104e1f  ()

Indirect leak of 9 byte(s) in 1 object(s) allocated from:
#0 0x7fb7aa213bf0 in operator new(unsigned long)
(/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc2bf0)
#1 0x7fb7a9cd3307 in __gnu_cxx::new_allocator::allocate(unsigned long, void const*)
/usr/include/c++/6/ext/new_allocator.h:104
#2 0x7fb7a9cd3307 in std::allocator_traits<std::allocator >::allocate(std::allocator&, unsigned long)
/usr/include/c++/6/bits/alloc_traits.h:436
#3 0x7fb7a9cd3307 in std::_Vector_base >::_M_allocate(unsigned long)
/usr/include/c++/6/bits/stl_vector.h:170
#4 0x7fb7a9cd3307 in unsigned char* std::vector
>::_M_allocate_and_copy<__gnu_cxx::__normal_iterator > > >(unsigned l
ong, __gnu_cxx::__normal_iterator > >,
__gnu_cxx::__normal_iterator > >) /usr/include/c
++/6/bits/stl_vector.h:1222
#5 0x7fb7a9cd3307 in std::vector >::operator=(std::vector
> const&) /usr/include/c++/6/bits/vector.tcc:195
#6 0x7fb7a9cd3307 in
SgAsmInstruction::set_raw_bytes(std::vector >)
/home/thomasdullien/dyninst-9.3.2/dataflowAPI/src/RoseImpl.C:1363
#7 0x602d4478  ()

Do these seem like real leaks? (This is not from trunk, but the 9.3
release).

Cheers,
Thomas

On Wed, Sep 27, 2017 at 3:59 PM, Thomas Dullien <thomasdull...@google.com>
wrote:

> Hey there,
>
> while tracing down some undesired behavior in Functionsimsearch, I
> compiled and
> ran everything under gcc6 and -fsanitize=address, which includes
> LeakSanitizer.
>
> I get a fairly large-ish number of reported leaks from Rose:
>
>
___
Dyninst-api mailing list
Dyninst-api@cs.wisc.edu
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api

[DynInst_API:] Question on potential memory leaks

[Thomas Dullien]

Hey there,

while tracing down some undesired behavior in Functionsimsearch, I compiled
and
ran everything under gcc6 and -fsanitize=address, which includes
LeakSanitizer.

I get a fairly large-ish number of reported leaks from Rose:
___
Dyninst-api mailing list
Dyninst-api@cs.wisc.edu
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api

Re: [DynInst_API:] where to find the code for handling switch() statements?

[Thomas Dullien]

Hey there,

ok, I identified one of my mistakes that led to poor performance on
switch()es -- I had marked the code section
as writeable, and the switch handling heuristics then declare all switch
jumptables invalid. Fixing this fixed
my issues with the switch code above :)

Cheers,
Thomas

On Tue, Aug 22, 2017 at 4:06 PM, Thomas Dullien <thomasdull...@google.com>
wrote:

> Hey there,
>
> regarding 64-bit PE binaries: I am providing the data to Dyninst myself,
> so anything that works "disassembly-wise" should work here, too. In the
> end, to Dyninst, the code is just a blob of assembly.
>
> Thanks a lot for the hints regarding the env var and the code, digging
> into it now :-)
>
> Cheers,
> Thomas
>
> On Tue, Aug 22, 2017 at 3:33 PM, Xiaozhu Meng <mxz...@gmail.com> wrote:
>
>> Hi Thomas,
>>
>> While Dyninst fully supports 64-bit ELF binaries, I don't think Dyninst
>> currently work with 64-bit PE binaries. I need to ask others to know how
>> much efforts are needed if you really want to do analyze 64-bit PE binaries.
>>
>> In terms of your 32-bit code example, the jump table construct looks very
>> primitive, so I am a little surprise that Dyninst currently failed to
>> analyze it.
>>
>> To debug this, you can first set "DYNINST_DEBUG_PARSING" to 1 and then
>> run your program again. This will dump the complete debugging log. In terms
>> of the code, you want to start with parseAPI/src/IndirectAnalyzer.C,
>> which performs the analysis of the jump tables. It contains two major
>> pieces: parseAPI/src/JumpTableFormatPred.C, which contains the code to
>> determine the jump table locations, jump table index variables, and other
>> format elements, and parseAPI/src/JumpTableIndexPred.C, which tries the
>> determine the value bound of the index variables.
>>
>> In your case, I am guessing that the problem is in JumpTableFormatPred.C.
>>
>> If you find it difficult to debug this by your own and if it is possible
>> to share this problematic binary with me, I can take a look at it.
>>
>> Thanks,
>>
>> --Xiaozhu
>>
>> On Tue, Aug 22, 2017 at 7:50 AM, Thomas Dullien <thomasdull...@google.com
>> > wrote:
>>
>>> Hey there,
>>>
>>> I gave the fork a try, but it does not seem to have handled the switch I
>>> encounter either. The construct looks
>>> as follows:
>>>
>>> .text:5A6E59FA pushebp
>>> .text:5A6E59FB mov ebp, esp
>>> .text:5A6E59FD sub esp, 18h
>>> .text:5A6E5A00 imuleax, [ebp+arg_4], 28h
>>> .text:5A6E5A04 pushebx
>>> .text:5A6E5A05 mov ebx, [ebp+arg_0]
>>> .text:5A6E5A08 pushesi
>>> .text:5A6E5A09 mov esi, ecx
>>> .text:5A6E5A0B mov [ebp+var_8], 17D7840h
>>> .text:5A6E5A12 add eax, ebx
>>> .text:5A6E5A14 mov [ebp+var_14], esi
>>> .text:5A6E5A17 mov [ebp+var_C], ebx
>>> .text:5A6E5A1A mov [ebp+var_18], eax
>>> .text:5A6E5A1D pushedi
>>> .text:5A6E5A1E cmp ebx, eax
>>> .text:5A6E5A20 jnb loc_5A6E608A
>>> .text:5A6E5A26 lea eax, [ebx+8]
>>> .text:5A6E5A29 mov ecx, esi
>>> .text:5A6E5A2B pusheax
>>> .text:5A6E5A2C call(..)
>>> .text:5A6E5A31 mov edi, eax
>>> .text:5A6E5A33 lea eax, [ebx+18h]
>>> .text:5A6E5A36 pusheax
>>> .text:5A6E5A37 call(...)
>>> .text:5A6E5A3C mov ecx, eax
>>> .text:5A6E5A3E mov eax, [ebx]
>>> .text:5A6E5A40 cmp eax, 36h; switch 55 cases
>>> .text:5A6E5A43 ja  loc_5A6E6095; jumptable
>>> 5A6E5A49 default case
>>> .text:5A6E5A49 jmp ds:off_5A6E609A[eax*4] ; switch
>>> jump
>>>
>>> Any advice on where in the dyninst codebase I should go digging for the
>>> switch handling code?
>>>
>>> Cheers,
>>> Thomas
>>>
>>> On Tue, Aug 22, 2017 at 1:26 PM, Thomas Dullien <
>>> thomasdull...@google.com> wrote:
>>>
>>>> Hey there,
>>>>
>>>> an example from 32-bit code where the default sw

Re: [DynInst_API:] where to find the code for handling switch() statements?

[Thomas Dullien]

Hey there,

I gave the fork a try, but it does not seem to have handled the switch I
encounter either. The construct looks
as follows:

.text:5A6E59FA pushebp
.text:5A6E59FB mov ebp, esp
.text:5A6E59FD sub esp, 18h
.text:5A6E5A00 imuleax, [ebp+arg_4], 28h
.text:5A6E5A04 pushebx
.text:5A6E5A05 mov ebx, [ebp+arg_0]
.text:5A6E5A08 pushesi
.text:5A6E5A09 mov esi, ecx
.text:5A6E5A0B mov [ebp+var_8], 17D7840h
.text:5A6E5A12 add eax, ebx
.text:5A6E5A14 mov [ebp+var_14], esi
.text:5A6E5A17 mov [ebp+var_C], ebx
.text:5A6E5A1A mov [ebp+var_18], eax
.text:5A6E5A1D pushedi
.text:5A6E5A1E cmp ebx, eax
.text:5A6E5A20 jnb loc_5A6E608A
.text:5A6E5A26 lea eax, [ebx+8]
.text:5A6E5A29 mov ecx, esi
.text:5A6E5A2B pusheax
.text:5A6E5A2C call(..)
.text:5A6E5A31 mov edi, eax
.text:5A6E5A33 lea eax, [ebx+18h]
.text:5A6E5A36 pusheax
.text:5A6E5A37 call(...)
.text:5A6E5A3C mov ecx, eax
.text:5A6E5A3E mov eax, [ebx]
.text:5A6E5A40 cmp eax, 36h; switch 55 cases
.text:5A6E5A43 ja  loc_5A6E6095; jumptable 5A6E5A49
default case
.text:5A6E5A49 jmp ds:off_5A6E609A[eax*4] ; switch jump

Any advice on where in the dyninst codebase I should go digging for the
switch handling code?

Cheers,
Thomas

On Tue, Aug 22, 2017 at 1:26 PM, Thomas Dullien <thomasdull...@google.com>
wrote:

> Hey there,
>
> an example from 32-bit code where the default switch handling fails:
>
> .text:00412990 sub esp, 50h
> .text:00412993 mov eax, ___security_cookie
> .text:00412998 xor eax, esp
> .text:0041299A mov [esp+50h+var_4], eax
> .text:0041299E mov edx, [esp+50h+arg_0]
> .text:004129A2 pushebx
> .text:004129A3 mov ebx, ecx
> .text:004129A5 lea eax, [edx-1]
> .text:004129A8 cmp eax, 6  ; switch 7 cases
> .text:004129AB ja  loc_412F7E  ; jumptable
> 004129B4 default case
> .text:004129B1 pushebp
> .text:004129B2 pushesi
> .text:004129B3 pushedi
> .text:004129B4 jmp ds:off_412F90[eax*4] ; switch jump
>
> Enough of this for the moment, though :-)) -- I will check your branch now
> :-)
>
> Cheers,
> Thomas
>
> On Tue, Aug 22, 2017 at 1:24 PM, Thomas Dullien <thomasdull...@google.com>
> wrote:
>
>> Hey there,
>>
>> I am back at work on this :-).
>>
>> A few questions:
>>  - Your fork is a fork of Dyninst 9 ?
>>  - Are there any things I need to be aware of when building it?
>>
>> The particular scenario I am dealing with right now is the following
>> construct (x86_64 disassembly of
>> Visual Studio compiled code).
>>
>> .text:00014004D970 mov [rsp+arg_8], edx
>> .text:00014004D974 mov [rsp+arg_0], rcx
>> .text:00014004D979 pushrdi
>> .text:00014004D97A sub rsp, 220h
>> .text:00014004D981 mov rdi, rsp
>> .text:00014004D984 mov ecx, 88h
>> .text:00014004D989 mov eax, 0h
>> .text:00014004D98E rep stosd
>> .text:00014004D990 mov rcx, [rsp+228h+arg_0]
>> .text:00014004D998 mov rax, cs:__security_cookie
>> .text:00014004D99F xor rax, rsp
>> .text:00014004D9A2 mov [rsp+228h+var_18], rax
>> .text:00014004D9AA mov eax, [rsp+228h+arg_8]
>> .text:00014004D9B1 mov [rsp+228h+var_80], eax
>> .text:00014004D9B8 mov eax, [rsp+228h+var_80]
>> .text:00014004D9BF dec eax
>> .text:00014004D9C1 mov [rsp+228h+var_80], eax
>> .text:00014004D9C8 cmp [rsp+228h+var_80], 5 ;
>> switch 6 cases
>> .text:00014004D9D0 ja  loc_14004EA48   ;
>> jumptable 00014004D9EF default case
>> .text:00014004D9D6 movsxd  rax, [rsp+228h+var_80]
>> .text:00

Re: [DynInst_API:] where to find the code for handling switch() statements?

[Thomas Dullien]

Hey there,

an example from 32-bit code where the default switch handling fails:

.text:00412990 sub esp, 50h
.text:00412993 mov eax, ___security_cookie
.text:00412998 xor eax, esp
.text:0041299A mov [esp+50h+var_4], eax
.text:0041299E mov edx, [esp+50h+arg_0]
.text:004129A2 pushebx
.text:004129A3 mov ebx, ecx
.text:004129A5 lea eax, [edx-1]
.text:004129A8 cmp eax, 6  ; switch 7 cases
.text:004129AB ja  loc_412F7E  ; jumptable 004129B4
default case
.text:004129B1 pushebp
.text:004129B2 pushesi
.text:004129B3 pushedi
.text:004129B4 jmp ds:off_412F90[eax*4] ; switch jump

Enough of this for the moment, though :-)) -- I will check your branch now
:-)

Cheers,
Thomas

On Tue, Aug 22, 2017 at 1:24 PM, Thomas Dullien <thomasdull...@google.com>
wrote:

> Hey there,
>
> I am back at work on this :-).
>
> A few questions:
>  - Your fork is a fork of Dyninst 9 ?
>  - Are there any things I need to be aware of when building it?
>
> The particular scenario I am dealing with right now is the following
> construct (x86_64 disassembly of
> Visual Studio compiled code).
>
> .text:00014004D970 mov [rsp+arg_8], edx
> .text:00014004D974 mov [rsp+arg_0], rcx
> .text:00014004D979 pushrdi
> .text:00014004D97A sub rsp, 220h
> .text:00014004D981 mov rdi, rsp
> .text:00014004D984 mov ecx, 88h
> .text:00014004D989 mov eax, 0h
> .text:00014004D98E rep stosd
> .text:00014004D990 mov rcx, [rsp+228h+arg_0]
> .text:00014004D998 mov rax, cs:__security_cookie
> .text:00014004D99F xor rax, rsp
> .text:00014004D9A2 mov [rsp+228h+var_18], rax
> .text:00014004D9AA mov eax, [rsp+228h+arg_8]
> .text:00014004D9B1 mov [rsp+228h+var_80], eax
> .text:00014004D9B8 mov eax, [rsp+228h+var_80]
> .text:00014004D9BF dec eax
> .text:00014004D9C1 mov [rsp+228h+var_80], eax
> .text:00014004D9C8 cmp [rsp+228h+var_80], 5 ;
> switch 6 cases
> .text:00014004D9D0 ja  loc_14004EA48   ; jumptable
> 00014004D9EF default case
> .text:00014004D9D6 movsxd  rax, [rsp+228h+var_80]
> .text:00014004D9DE lea rcx, cs:14000h
> .text:00014004D9E5 mov eax, ds:(off_14004EA70 -
> 14000h)[rcx+rax*4]
> .text:00014004D9EC add rax, rcx
> .text:00014004D9EF jmp rax ; switch
> jump
> .text:00014004D9F1 ; ------
> -
>
> Cheers,
> Thomas
>
> On Tue, Jun 13, 2017 at 4:35 PM, Thomas Dullien <thomasdull...@google.com>
> wrote:
>
>> Hey there,
>>
>> excellent, thanks for your quick response :-) I will give your fork a try
>> in the next 2-3 days -- I am currently
>> at a conference and hence won't have time to try it today :-)
>>
>> Cheers,
>> Thomas
>>
>> On Tue, Jun 13, 2017 at 10:30 AM, Xiaozhu Meng <mxz...@gmail.com> wrote:
>>
>>> Hi Thomas,
>>>
>>> I am working with an improved jump table analysis. Its prototype is
>>> available at my Dyninst fork (https://github.com/mxz297/dyn
>>> inst/tree/jump_table_multi_slices). This improved version should be
>>> merged back to mainstream Dyninst in the near future. Could you try my
>>> version to see whether it solves your problem? If the problem remains,
>>> could you provide me the problematic binary so that I can further improve
>>> my code?
>>>
>>> Thanks,
>>>
>>> --Xiaozhu
>>>
>>> On Tue, Jun 13, 2017 at 7:25 AM, Thomas Dullien <
>>> thomasdull...@google.com> wrote:
>>>
>>>> Hey all,
>>>>
>>>> I am using DynInst for a small project that helps search for similar
>>>> flowgraph in a search index (https://www.github.com/thomas
>>>> dullien/functionsimsearch)
>>>> and noticed that most switch statements that it encounters are not
>>>> handled properly (e.g. the control flow reconstruction fails to resolve
>>>> the swi

Re: [DynInst_API:] where to find the code for handling switch() statements?

[Thomas Dullien]

Hey there,

I am back at work on this :-).

A few questions:
 - Your fork is a fork of Dyninst 9 ?
 - Are there any things I need to be aware of when building it?

The particular scenario I am dealing with right now is the following
construct (x86_64 disassembly of
Visual Studio compiled code).

.text:00014004D970 mov [rsp+arg_8], edx
.text:00014004D974 mov [rsp+arg_0], rcx
.text:00014004D979 pushrdi
.text:00014004D97A sub rsp, 220h
.text:00014004D981 mov rdi, rsp
.text:00014004D984 mov ecx, 88h
.text:00014004D989 mov eax, 0h
.text:00014004D98E rep stosd
.text:00014004D990 mov rcx, [rsp+228h+arg_0]
.text:00014004D998 mov rax, cs:__security_cookie
.text:00014004D99F xor rax, rsp
.text:00014004D9A2 mov [rsp+228h+var_18], rax
.text:00014004D9AA mov eax, [rsp+228h+arg_8]
.text:00014004D9B1 mov [rsp+228h+var_80], eax
.text:00014004D9B8 mov eax, [rsp+228h+var_80]
.text:00014004D9BF dec eax
.text:00014004D9C1 mov [rsp+228h+var_80], eax
.text:00014004D9C8 cmp [rsp+228h+var_80], 5 ;
switch 6 cases
.text:00014004D9D0 ja  loc_14004EA48   ; jumptable
00014004D9EF default case
.text:00014004D9D6 movsxd  rax, [rsp+228h+var_80]
.text:00014004D9DE lea rcx, cs:14000h
.text:00014004D9E5 mov eax, ds:(off_14004EA70 -
14000h)[rcx+rax*4]
.text:00014004D9EC add rax, rcx
.text:00014004D9EF jmp rax ; switch jump
.text:00014004D9F1 ;
---

Cheers,
Thomas

On Tue, Jun 13, 2017 at 4:35 PM, Thomas Dullien <thomasdull...@google.com>
wrote:

> Hey there,
>
> excellent, thanks for your quick response :-) I will give your fork a try
> in the next 2-3 days -- I am currently
> at a conference and hence won't have time to try it today :-)
>
> Cheers,
> Thomas
>
> On Tue, Jun 13, 2017 at 10:30 AM, Xiaozhu Meng <mxz...@gmail.com> wrote:
>
>> Hi Thomas,
>>
>> I am working with an improved jump table analysis. Its prototype is
>> available at my Dyninst fork (https://github.com/mxz297/dyn
>> inst/tree/jump_table_multi_slices). This improved version should be
>> merged back to mainstream Dyninst in the near future. Could you try my
>> version to see whether it solves your problem? If the problem remains,
>> could you provide me the problematic binary so that I can further improve
>> my code?
>>
>> Thanks,
>>
>> --Xiaozhu
>>
>> On Tue, Jun 13, 2017 at 7:25 AM, Thomas Dullien <thomasdull...@google.com
>> > wrote:
>>
>>> Hey all,
>>>
>>> I am using DynInst for a small project that helps search for similar
>>> flowgraph in a search index (https://www.github.com/thomas
>>> dullien/functionsimsearch)
>>> and noticed that most switch statements that it encounters are not
>>> handled properly (e.g. the control flow reconstruction fails to resolve
>>> the switch targets).
>>>
>>> Where in the source code should I go looking for the relevant code?
>>> I'd love to have a look around to see if it can be improved.
>>>
>>> Cheers,
>>> Thomas
>>>
>>> ___
>>> Dyninst-api mailing list
>>> Dyninst-api@cs.wisc.edu
>>> https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api
>>>
>>>
>>
>
___
Dyninst-api mailing list
Dyninst-api@cs.wisc.edu
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api

Re: [DynInst_API:] Concurrent calls to getInsns

[Thomas Dullien]

Hey there,

great to hear :). I am mostly using it to experiment with some code
similarity work (https://github.com/google/functionsimsearch),
but multithreading improvements are clearly always welcome :)

Cheers,
Thomas

On Mon, Jul 24, 2017 at 8:32 PM, Barton Miller <b...@cs.wisc.edu> wrote:

> Thomas,
>
> Note that Bill is busily working on heavily multi-threading parts of the
> Dyninst tookits, to improve parse and analysis time.  Some of that is
> already in place, but lots more to come.
>
> --bart
>
> On 7/24/2017 1:28 PM, Thomas Dullien wrote:
>
> Hey there,
>
> ok cool :-) -- I think the statement "assume thread-unsafe for the moment"
> is already quite helpful, fixed my
> code accordingly. I guess I will just wait for new releases & keep my eyes
> peeled for changes in that area? :)
>
> Cheers,
> Thomas
>
> On Mon, Jul 24, 2017 at 7:40 PM, Bill Williams <b...@cs.wisc.edu> wrote:
>
>> This is an area with some active work occurring. However, it is best to
>> assume that interfaces are thread-unsafe in the current releases and
>> require external locking.
>>
>> If you need guidance on how best to manage concurrent access to Dyninst
>> data in the meantime, let us know; the proper strategy depends to some
>> extent on your use case.
>>
>> --bw
>>
>> 
>> From: Dyninst-api <dyninst-api-boun...@cs.wisc.edu> on behalf of Thomas
>> Dullien <thomasdull...@google.com>
>> Sent: Monday, July 24, 2017 5:16 AM
>> To: dyninst-api
>> Subject: [DynInst_API:] Concurrent calls to getInsns
>>
>> Hey there,
>>
>> I am observing data races and inconsistent results when multiple
>> threads run through the following code:
>>
>> Dyninst::ParseAPI::Block::Insns block_instructions;
>> block->getInsns(block_instructions);
>> for (const auto& instruction : block_instructions) {  }
>>
>> Are Dyninst functions thread-unsafe by default, or what is the best way of
>> finding out which ones are safe and which ones aren't ? :-)
>>
>> Cheers,
>> Thomas
>> PS: Thanks for the awesome tool ! :)
>>
>
>
>
> ___
> Dyninst-api mailing 
> listdyninst-...@cs.wisc.eduhttps://lists.cs.wisc.edu/mailman/listinfo/dyninst-api
>
>
>
> ___
> Dyninst-api mailing list
> Dyninst-api@cs.wisc.edu
> https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api
>
___
Dyninst-api mailing list
Dyninst-api@cs.wisc.edu
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api

Re: [DynInst_API:] Concurrent calls to getInsns

[Thomas Dullien]

Hey there,

ok cool :-) -- I think the statement "assume thread-unsafe for the moment"
is already quite helpful, fixed my
code accordingly. I guess I will just wait for new releases & keep my eyes
peeled for changes in that area? :)

Cheers,
Thomas

On Mon, Jul 24, 2017 at 7:40 PM, Bill Williams <b...@cs.wisc.edu> wrote:

> This is an area with some active work occurring. However, it is best to
> assume that interfaces are thread-unsafe in the current releases and
> require external locking.
>
> If you need guidance on how best to manage concurrent access to Dyninst
> data in the meantime, let us know; the proper strategy depends to some
> extent on your use case.
>
> --bw
>
> 
> From: Dyninst-api <dyninst-api-boun...@cs.wisc.edu> on behalf of Thomas
> Dullien <thomasdull...@google.com>
> Sent: Monday, July 24, 2017 5:16 AM
> To: dyninst-api
> Subject: [DynInst_API:] Concurrent calls to getInsns
>
> Hey there,
>
> I am observing data races and inconsistent results when multiple
> threads run through the following code:
>
> Dyninst::ParseAPI::Block::Insns block_instructions;
> block->getInsns(block_instructions);
> for (const auto& instruction : block_instructions) {  }
>
> Are Dyninst functions thread-unsafe by default, or what is the best way of
> finding out which ones are safe and which ones aren't ? :-)
>
> Cheers,
> Thomas
> PS: Thanks for the awesome tool ! :)
>
___
Dyninst-api mailing list
Dyninst-api@cs.wisc.edu
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api

[DynInst_API:] Concurrent calls to getInsns

[Thomas Dullien]

Hey there,

I am observing data races and inconsistent results when multiple
threads run through the following code:

Dyninst::ParseAPI::Block::Insns block_instructions;
block->getInsns(block_instructions);
for (const auto& instruction : block_instructions) {  }

Are Dyninst functions thread-unsafe by default, or what is the best way of
finding out which ones are safe and which ones aren't ? :-)

Cheers,
Thomas
PS: Thanks for the awesome tool ! :)
___
Dyninst-api mailing list
Dyninst-api@cs.wisc.edu
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api

Re: [DynInst_API:] where to find the code for handling switch() statements?

[Thomas Dullien]

Hey there,

excellent, thanks for your quick response :-) I will give your fork a try
in the next 2-3 days -- I am currently
at a conference and hence won't have time to try it today :-)

Cheers,
Thomas

On Tue, Jun 13, 2017 at 10:30 AM, Xiaozhu Meng <mxz...@gmail.com> wrote:

> Hi Thomas,
>
> I am working with an improved jump table analysis. Its prototype is
> available at my Dyninst fork (https://github.com/mxz297/
> dyninst/tree/jump_table_multi_slices). This improved version should be
> merged back to mainstream Dyninst in the near future. Could you try my
> version to see whether it solves your problem? If the problem remains,
> could you provide me the problematic binary so that I can further improve
> my code?
>
> Thanks,
>
> --Xiaozhu
>
> On Tue, Jun 13, 2017 at 7:25 AM, Thomas Dullien <thomasdull...@google.com>
> wrote:
>
>> Hey all,
>>
>> I am using DynInst for a small project that helps search for similar
>> flowgraph in a search index (https://www.github.com/thomas
>> dullien/functionsimsearch)
>> and noticed that most switch statements that it encounters are not
>> handled properly (e.g. the control flow reconstruction fails to resolve
>> the switch targets).
>>
>> Where in the source code should I go looking for the relevant code?
>> I'd love to have a look around to see if it can be improved.
>>
>> Cheers,
>> Thomas
>>
>> ___
>> Dyninst-api mailing list
>> Dyninst-api@cs.wisc.edu
>> https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api
>>
>>
>
___
Dyninst-api mailing list
Dyninst-api@cs.wisc.edu
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api

[DynInst_API:] where to find the code for handling switch() statements?

[Thomas Dullien]

Hey all,

I am using DynInst for a small project that helps search for similar
flowgraph in a search index (
https://www.github.com/thomasdullien/functionsimsearch)
and noticed that most switch statements that it encounters are not
handled properly (e.g. the control flow reconstruction fails to resolve
the switch targets).

Where in the source code should I go looking for the relevant code?
I'd love to have a look around to see if it can be improved.

Cheers,
Thomas
___
Dyninst-api mailing list
Dyninst-api@cs.wisc.edu
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api