Re: [Kgdb-bugreport] Problem getting kgdb to read kernel symbols. addresses shifted?
Vivek Goyal wrote: On Fri, Sep 28, 2007 at 05:40:33PM -0600, Eric W. Biederman wrote: Derek Atkins [EMAIL PROTECTED] writes: Well, gdb agrees with System.map, so I'm sure that gdb itself is okay. It's certainly possible that that the kgdb stub is weird, but /proc/kallsyms doesn't match System.map, and THAT'S what's confusing me most of all. Ok. So we must have a relocatable kernel that figures it has been relocated. Interesting. What is your bootloader? What is your kernel version? What is your kernel config? The only time I would expect to see what you are seeing is if you are debugging the kdump kernel, which doesn't sound like the case. If we actually have a truly offset kernel then while things may not be perfect this is at least expected. I don't think I have heard of anyone handling this case very well. Hi Eric and others, I think we might be running into the issues because i386, FC7 relocatable kernel has been compiled for 16MB physical address but effectively it runs at 4MB physical address. So kernel does not run at compiled address and any kind of debugging tools reading symbol address from System.map or rom vmlinux will fail as run time symbol addresses are different. /proc/kallsyms should help though. This is one problem with shift in run time virtual address while relocating the kernel. We should be running kernel at compiled address to be able to debug it. Or enable any tools to parse /proc/kallsyms to read the shift in symbol addresses and adjust accordingly. Thanks Vivek Right, crash was updated in version 4.0-4.5 to allow the use of /proc/kallsyms as an alternative to the System.map file, as well as adding a new --reloc command line argument. After bringing up the vmlinux file in gdb (with the wrong addresses), all of the minimal_symbol data structures in the gdb module are back-patched with the /proc/kallsyms values: http://people.redhat.com/anderson/crash.changelog.html#4_0_4_5 It seems the benefit of configuring the kernel that way is debatable, and I will do all I can to convince the RHEL-6 and beyond kernel maintainers from doing it that way in the future. But Fedora goes its own way. Seems totally lame to issue a bogus System.map file though... Dave - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Kgdb-bugreport mailing list Kgdb-bugreport@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kgdb-bugreport
Re: [Kgdb-bugreport] Problem getting kgdb to read kernel symbols. addresses shifted?
Derek Atkins wrote: Dave Anderson [EMAIL PROTECTED] writes: Hi Eric and others, I think we might be running into the issues because i386, FC7 relocatable kernel has been compiled for 16MB physical address but effectively it runs at 4MB physical address. So kernel does not run at compiled address and any kind of debugging tools reading symbol address from System.map or rom vmlinux will fail as run time symbol addresses are different. /proc/kallsyms should help though. This is one problem with shift in run time virtual address while relocating the kernel. We should be running kernel at compiled address to be able to debug it. Or enable any tools to parse /proc/kallsyms to read the shift in symbol addresses and adjust accordingly. Thanks Vivek Right, crash was updated in version 4.0-4.5 to allow the use of /proc/kallsyms as an alternative to the System.map file, as well as adding a new --reloc command line argument. After bringing up the vmlinux file in gdb (with the wrong addresses), all of the minimal_symbol data structures in the gdb module are back-patched with the /proc/kallsyms values: http://people.redhat.com/anderson/crash.changelog.html#4_0_4_5 It seems the benefit of configuring the kernel that way is debatable, and I will do all I can to convince the RHEL-6 and beyond kernel maintainers from doing it that way in the future. But Fedora goes its own way. Seems totally lame to issue a bogus System.map file though... Well, this is kgdb, so /proc/kallsyms is on the target machine, not the host machine. So, 'gdb' cannot read /proc/kallsyms, because, well, it's not local. Well, you could just grab a copy of it, although I don't know what gdb alone would be able to do with it... However, I AM building my own kernel -- so I can certainly reconfigure it as necessary. What do I need to do to reconfigure my kernel to run at the same place it was built for? I.e., what's changing the runtime from 16M to 4M and how do I make it consistent? Thanks! Configure the kernel with CONFIG_PHYSICAL_START less than or equal to CONFIG_PHYSICAL_ALIGN. Upon rebuilding my FC7 kernel with CONFIG_PHYSICAL_START changed from 16MB to 1MB, with CONFIG_PHYSICAL_ALIGN left at 4MB, i.e.: CONFIG_PHYSICAL_START=0x10 CONFIG_PHYSICAL_ALIGN=0x40 The kernel gets compiled for, and runs at, a 4MB physical address: $ nm -Bn vmlinux | grep ^c04 c040 T _text c040 T startup_32 c0401000 T startup_32_smp c0401080 t checkCPUtype c0401101 t is486 ... Setting both of them to 0x40 also works. Dave - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Kgdb-bugreport mailing list Kgdb-bugreport@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kgdb-bugreport
Re: [Kgdb-bugreport] Problem getting kgdb to read kernel symbols. addresses shifted?
I have not looked at the fedora kernels, but it would appear that if
there was a shift that some part of the loader or some kernel patch has
done a runtime relocation.
If you knew for absolutely certain that everything was relocated with an
offset, then you could simply load the symbols with an offset too. You
are not going to be able to easily debug kernel modules of course
without patching gdb to have the kernel module support.
Jason.
Derek Atkins wrote:
Hi everyone,
I'm using Fedora 7; I've backported the 2.6.23-rc8-mm1 broken-out kgdb
patches (git-kgdb.patch, git-kgdb-fixup.patch,
kgdb-fix-docbook-and-kernel-doc-typos.patch, and
kgdb-fix-help-text.patch) to the Fedora 2.6.22.5-76 kernel
and I can connect just fine from my host system to the
target running in vmware.
Unfortunately, gdb shows no symbols:
[EMAIL PROTECTED] ~]$ gdb /vmlinux.kgdb
GNU gdb Red Hat Linux (6.6-15.fc7rh)
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type show copying to see the conditions.
There is absolutely no warranty for GDB. Type show warranty for details.
This GDB was configured as i386-redhat-linux-gnu...
Using host libthread_db library /lib/libthread_db.so.1.
(gdb) target remote /dev/pts/18
Remote debugging using /dev/pts/18
warning: shared library handler failed to enable breakpoint
0xc04579aa in ?? ()
(gdb) bt
#0 0xc04579aa in ?? ()
#1 0xc07a2fc8 in ?? ()
#2 0xc055fe3e in ?? ()
#3 0xc07a2fe0 in ?? ()
#4 0xc04597f6 in ?? ()
#5 0x0004 in ?? ()
#6 0xc072a000 in ?? ()
#7 0x0004 in ?? ()
#8 0xdfb247e0 in ?? ()
#9 0xc07a2ff8 in ?? ()
#10 0xc045ab73 in ?? ()
#11 0xc072a02c in ?? ()
#12 0xc0736f54 in ?? ()
#13 0x0004 in ?? ()
#14 0xc045aaad in ?? ()
#15 0xc0736f70 in ?? ()
#16 0xc0407309 in ?? ()
#17 0x in ?? ()
(gdb) p sys_close
$1 = {long int (unsigned int)} 0xc107e341 sys_close
(gdb)
This matches what's in the System.map file:
[EMAIL PROTECTED] ~] grep sys_close .../System.map
c107e341 T sys_close
c12deb58 r __ksymtab_sys_close
c12e4918 r __kcrctab_sys_close
c12ea703 r __kstrtab_sys_close
However, the system running this kernel shows a different value:
[EMAIL PROTECTED] #] grep sys_close /proc/kallsyms
c047e341 T sys_close
c06deb58 r __ksymtab_sys_close
c06e4918 r __kcrctab_sys_close
c06ea703 r __kstrtab_sys_close
I find it interesting that the symbols are EXACTLY 0x60 different,
but I haven't been able to track down this difference. Am I doing
something wrong? Is there some particular kernel configuration I need
to set for my kernel? Why are the symbols in the running kernel at
different addresses than the underlying vmlinux?
(I'll also note that I see a similar skew on my non-rebuilt Fedora
kernel, comparing System.map to /proc/kallsyms, so it's probably not
specifically a kgdb issue, it could be a Fedora issue.. but I'd still
like help in working around this disparity).
Thanks in advance for any help.
-derek
-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
Kgdb-bugreport mailing list
Kgdb-bugreport@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/kgdb-bugreport
Re: [Kgdb-bugreport] Problem getting kgdb to read kernel symbols. addresses shifted?
Derek Atkins wrote: Hi, Jason Wessel [EMAIL PROTECTED] writes: I have not looked at the fedora kernels, but it would appear that if there was a shift that some part of the loader or some kernel patch has done a runtime relocation. Well, it appears to be the case, but I've only tested that one symbol. I could go randomly sample some other symbols, too, if that would help. It certainly seems weird to me, but it does seem consistent. :( If you knew for absolutely certain that everything was relocated with an offset, then you could simply load the symbols with an offset too. You Um, okay. How do I do that? My GDB Fu is weak here; how do I tell gdb that the symbols in vmlinux are all offset? Or how do I manipulate the vmlinux binary to offset the symbols? Start gdb with no file. And do something like: add-symbol-file vmlinux 0xBFA0 I doubt this will fix your problem if the text and data section were dynamically changed or changed as a result of using something like kexec/kdump. are not going to be able to easily debug kernel modules of course without patching gdb to have the kernel module support. You can elect to make manual mappings, but is automatically taken care of if you port the changes to gdb which are in the source forge kgdb repository. Jason. - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Kgdb-bugreport mailing list Kgdb-bugreport@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kgdb-bugreport
Re: [Kgdb-bugreport] Problem getting kgdb to read kernel symbols. addresses shifted?
Quoting Jason Wessel [EMAIL PROTECTED]: Um, okay. How do I do that? My GDB Fu is weak here; how do I tell gdb that the symbols in vmlinux are all offset? Or how do I manipulate the vmlinux binary to offset the symbols? Start gdb with no file. And do something like: add-symbol-file vmlinux 0xBFA0 Um, where did you get 0xBFA from? Unfortunately this didn't work at all. I would think to get my numbers to work I'd need to use 0xC040 to get sys_close to appear at 0xc047d341. And viola, that seems to work! Or at least I got a reasonable breakpoint from the 'target remote' I doubt this will fix your problem if the text and data section were dynamically changed or changed as a result of using something like kexec/kdump. Thanks for the warning -- I'll keep that in mind. I have no idea if the kernel build uses kexec/kdump. are not going to be able to easily debug kernel modules of course without patching gdb to have the kernel module support. You can elect to make manual mappings, but is automatically taken care of if you port the changes to gdb which are in the source forge kgdb repository. I'll take a look... It would be nice if these patches got fed upstream. Jason. -derek -- Derek Atkins, SB '93 MIT EE, SM '95 MIT Media Laboratory Member, MIT Student Information Processing Board (SIPB) URL: http://web.mit.edu/warlord/PP-ASEL-IA N1NWH [EMAIL PROTECTED]PGP key available - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Kgdb-bugreport mailing list Kgdb-bugreport@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kgdb-bugreport
Re: [Kgdb-bugreport] Problem getting kgdb to read kernel symbols. addresses shifted?
Derek Atkins wrote: Quoting Jason Wessel [EMAIL PROTECTED]: Um, okay. How do I do that? My GDB Fu is weak here; how do I tell gdb that the symbols in vmlinux are all offset? Or how do I manipulate the vmlinux binary to offset the symbols? Start gdb with no file. And do something like: add-symbol-file vmlinux 0xBFA0 Um, where did you get 0xBFA from? Unfortunately this didn't work at all. I would think to get my numbers to work I'd need to use 0xC040 to get sys_close to appear at 0xc047d341. And viola, that seems to work! Or at least I got a reasonable breakpoint from the 'target remote' It was an example. You had previously stated it was an offset of 0x60 so it was a subtraction of 0x60 from 0xc000. Jaosn. - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Kgdb-bugreport mailing list Kgdb-bugreport@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kgdb-bugreport
Re: [Kgdb-bugreport] Problem getting kgdb to read kernel symbols. addresses shifted?
Pete/Piet Delaney [EMAIL PROTECTED] writes: Jason Wessel wrote: Shouldn't crash, kdump and kgdb take into consideration a shift in the kernel so that gdb works normally? Seems that having the kgdb stub knowledgeable of a shift in the kernel might be easy to compensate for. Perhaps just mapping all reads and writes that lie within the original kernel segments to the shifted addresses. Good question. I remember hearing some oddball reports about something like this. My memory says what little tracking that happened pointed in the direction of a confused gdb. In particular unless you deliberately run a kernel at a different address linux kernels should run at the addresses they were built for so there should be nothing that has to be done. I'm running on foggy memories so something may have changed since last time I looked. Eric -piet Derek Atkins wrote: Quoting Jason Wessel [EMAIL PROTECTED]: Um, okay. How do I do that? My GDB Fu is weak here; how do I tell gdb that the symbols in vmlinux are all offset? Or how do I manipulate the vmlinux binary to offset the symbols? Start gdb with no file. And do something like: add-symbol-file vmlinux 0xBFA0 Um, where did you get 0xBFA from? Unfortunately this didn't work at all. I would think to get my numbers to work I'd need to use 0xC040 to get sys_close to appear at 0xc047d341. And viola, that seems to work! Or at least I got a reasonable breakpoint from the 'target remote' It was an example. You had previously stated it was an offset of 0x60 so it was a subtraction of 0x60 from 0xc000. Jaosn. - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Kgdb-bugreport mailing list Kgdb-bugreport@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kgdb-bugreport - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Kgdb-bugreport mailing list Kgdb-bugreport@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kgdb-bugreport
Re: [Kgdb-bugreport] Problem getting kgdb to read kernel symbols. addresses shifted?
Derek Atkins [EMAIL PROTECTED] writes: Well, gdb agrees with System.map, so I'm sure that gdb itself is okay. It's certainly possible that that the kgdb stub is weird, but /proc/kallsyms doesn't match System.map, and THAT'S what's confusing me most of all. Ok. So we must have a relocatable kernel that figures it has been relocated. Interesting. What is your bootloader? What is your kernel version? What is your kernel config? The only time I would expect to see what you are seeing is if you are debugging the kdump kernel, which doesn't sound like the case. If we actually have a truly offset kernel then while things may not be perfect this is at least expected. I don't think I have heard of anyone handling this case very well. Which was how long ago? ;) Long enough ago that I don't remember when ;) Eric - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Kgdb-bugreport mailing list Kgdb-bugreport@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kgdb-bugreport
[Kgdb-bugreport] Problem getting kgdb to read kernel symbols. addresses shifted?
Hi everyone,
I'm using Fedora 7; I've backported the 2.6.23-rc8-mm1 broken-out kgdb
patches (git-kgdb.patch, git-kgdb-fixup.patch,
kgdb-fix-docbook-and-kernel-doc-typos.patch, and
kgdb-fix-help-text.patch) to the Fedora 2.6.22.5-76 kernel
and I can connect just fine from my host system to the
target running in vmware.
Unfortunately, gdb shows no symbols:
[EMAIL PROTECTED] ~]$ gdb /vmlinux.kgdb
GNU gdb Red Hat Linux (6.6-15.fc7rh)
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type show copying to see the conditions.
There is absolutely no warranty for GDB. Type show warranty for details.
This GDB was configured as i386-redhat-linux-gnu...
Using host libthread_db library /lib/libthread_db.so.1.
(gdb) target remote /dev/pts/18
Remote debugging using /dev/pts/18
warning: shared library handler failed to enable breakpoint
0xc04579aa in ?? ()
(gdb) bt
#0 0xc04579aa in ?? ()
#1 0xc07a2fc8 in ?? ()
#2 0xc055fe3e in ?? ()
#3 0xc07a2fe0 in ?? ()
#4 0xc04597f6 in ?? ()
#5 0x0004 in ?? ()
#6 0xc072a000 in ?? ()
#7 0x0004 in ?? ()
#8 0xdfb247e0 in ?? ()
#9 0xc07a2ff8 in ?? ()
#10 0xc045ab73 in ?? ()
#11 0xc072a02c in ?? ()
#12 0xc0736f54 in ?? ()
#13 0x0004 in ?? ()
#14 0xc045aaad in ?? ()
#15 0xc0736f70 in ?? ()
#16 0xc0407309 in ?? ()
#17 0x in ?? ()
(gdb) p sys_close
$1 = {long int (unsigned int)} 0xc107e341 sys_close
(gdb)
This matches what's in the System.map file:
[EMAIL PROTECTED] ~] grep sys_close .../System.map
c107e341 T sys_close
c12deb58 r __ksymtab_sys_close
c12e4918 r __kcrctab_sys_close
c12ea703 r __kstrtab_sys_close
However, the system running this kernel shows a different value:
[EMAIL PROTECTED] #] grep sys_close /proc/kallsyms
c047e341 T sys_close
c06deb58 r __ksymtab_sys_close
c06e4918 r __kcrctab_sys_close
c06ea703 r __kstrtab_sys_close
I find it interesting that the symbols are EXACTLY 0x60 different,
but I haven't been able to track down this difference. Am I doing
something wrong? Is there some particular kernel configuration I need
to set for my kernel? Why are the symbols in the running kernel at
different addresses than the underlying vmlinux?
(I'll also note that I see a similar skew on my non-rebuilt Fedora
kernel, comparing System.map to /proc/kallsyms, so it's probably not
specifically a kgdb issue, it could be a Fedora issue.. but I'd still
like help in working around this disparity).
Thanks in advance for any help.
-derek
--
Derek Atkins, SB '93 MIT EE, SM '95 MIT Media Laboratory
Member, MIT Student Information Processing Board (SIPB)
URL: http://web.mit.edu/warlord/PP-ASEL-IA N1NWH
[EMAIL PROTECTED]PGP key available
-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
Kgdb-bugreport mailing list
Kgdb-bugreport@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/kgdb-bugreport
