Re: kernel since 5.6 do not boot anymore on Apple PowerBook
On Thu, Aug 27, 2020 at 12:39:06PM +0200, Christophe Leroy wrote: > Hi, > > Le 27/08/2020 à 10:28, Giuseppe Sacco a écrit : > > Il giorno gio, 27/08/2020 alle 09.46 +0200, Giuseppe Sacco ha scritto: > > > Il giorno gio, 27/08/2020 alle 00.28 +0200, Giuseppe Sacco ha scritto: > > > > Hello Christophe, > > > > > > > > Il giorno mer, 26/08/2020 alle 15.53 +0200, Christophe Leroy ha > > > > scritto: > > > > [...] > > > > > If there is no warning, then the issue is something else, bad luck. > > > > > > > > > > Could you increase the loglevel and try again both with and without > > > > > VMAP_STACK ? Maybe we'll get more information on where it stops. > > > > > > > > The problem is related to the CPU frequency changes. This is where the > > > > system stop: cpufreq get the CPU frequency limits and then start the > > > > default governor (performance) and then calls function > > > > cpufreq_gov_performance_limits() that never returns. > > > > > > > > Rebuilding after enabling pr_debug for cpufreq.c, I've got some more > > > > lines of output: > > > > > > > > cpufreq: setting new policy for CPU 0: 667000 - 867000 kHz > > > > cpufreq: new min and max freqs are 667000 - 867000 kHz > > > > cpufreq: governor switch > > > > cpufreq: cpufreq_init_governor: for CPU 0 > > > > cpufreq: cpufreq_start_governor: for CPU 0 > > > > cpufreq: target for CPU 0: 867000 kHz, relation 1, requested 867000 kHz > > > > cpufreq: __target_index: cpu: 0, oldfreq: 667000, new freq: 867000 > > > > cpufreq: notification 0 of frequency transition to 867000 kHz > > > > cpufreq: saving 133328 as reference value for loops_per_jiffy; freq is > > > > 667000 kHz > > > > > > > > no more lines are printed. I think this output only refers to the > > > > notification sent prior to the frequency change. > > > > > > > > I was thinking that selecting a governor that would run at 667mHz would > > > > probably skip the problem. I added cpufreq.default_governor=powersave > > > > to the command line parameters but it did not work: the selected > > > > governor was still performance and the system crashed. > > > > > > I kept following the thread and found that CPU frequency is changed in > > > function pmu_set_cpu_speed() in file drivers/cpufreq/pmac32-cpufreq.c. > > > As first thing, the function calls the macro preempt_disable() and this > > > is where it stops. > > > > Sorry, I made a mistake. The real problem is down, on the same > > function, when it calls low_sleep_handler(). This is where the problem > > probably is. > > > > > Great, you spotted the problem. > > I see what it is, it is in low_sleep_handler() in > arch/powerpc/platforms/powermac/sleep.S > > All critical registers are saved on the stack. At restore, they are restore > BEFORE re-enabling MMU (because they are needed for that). But when we have > VMAP_STACK, the stack can hardly be accessed without the MMU enabled. > tophys() doesn't work for virtual stack addresses. > > Therefore, the low_sleep_handler() has to be reworked for using an area in > the linear mem instead of the stack. Actually there is already one, at the end of sleep.S, there is a 4 byte area called sleep_storage. It should be enough to move there the CPU state which has to be restored before the MMU is enabled: - SDR1 - BATs - SPRG (they might stay on the stack by reordering the code unless I miss something) Note that save_cpu_setup/restore_cpu_setup also use another static storage area to save other SPRs. Using static area would not work on SMP, but all PPC based Apple laptops are single core and single thread. Gabriel
Re: kernel since 5.6 do not boot anymore on Apple PowerBook
Le 27/08/2020 à 16:37, Giuseppe Sacco a écrit : Il giorno gio, 27/08/2020 alle 12.39 +0200, Christophe Leroy ha scritto: Hi, Le 27/08/2020 à 10:28, Giuseppe Sacco a écrit : [...] Sorry, I made a mistake. The real problem is down, on the same function, when it calls low_sleep_handler(). This is where the problem probably is. Great, you spotted the problem. I see what it is, it is in low_sleep_handler() in arch/powerpc/platforms/powermac/sleep.S All critical registers are saved on the stack. At restore, they are restore BEFORE re-enabling MMU (because they are needed for that). But when we have VMAP_STACK, the stack can hardly be accessed without the MMU enabled. tophys() doesn't work for virtual stack addresses. Therefore, the low_sleep_handler() has to be reworked for using an area in the linear mem instead of the stack. I am sorry, but I don't know how to fix it. Should I open a bug for tracking this problem? Yes please, at https://github.com/linuxppc/issues/issues In the meantime, I have sent a patch to disable CONFIG_VMAP_STACK when CONFIG_ADB_PMU is selected until this is fixed. Have you tried without CONFIG_ADB_PMU ? Or does it make no sense ? Christophe
Re: kernel since 5.6 do not boot anymore on Apple PowerBook
Il giorno gio, 27/08/2020 alle 12.39 +0200, Christophe Leroy ha scritto: > Hi, > > Le 27/08/2020 à 10:28, Giuseppe Sacco a écrit : [...] > > Sorry, I made a mistake. The real problem is down, on the same > > function, when it calls low_sleep_handler(). This is where the problem > > probably is. > > Great, you spotted the problem. > > I see what it is, it is in low_sleep_handler() in > arch/powerpc/platforms/powermac/sleep.S > > All critical registers are saved on the stack. At restore, they are > restore BEFORE re-enabling MMU (because they are needed for that). But > when we have VMAP_STACK, the stack can hardly be accessed without the > MMU enabled. tophys() doesn't work for virtual stack addresses. > > Therefore, the low_sleep_handler() has to be reworked for using an area > in the linear mem instead of the stack. I am sorry, but I don't know how to fix it. Should I open a bug for tracking this problem? Thank you, Giuseppe
Re: kernel since 5.6 do not boot anymore on Apple PowerBook
Hi, Le 27/08/2020 à 10:28, Giuseppe Sacco a écrit : Il giorno gio, 27/08/2020 alle 09.46 +0200, Giuseppe Sacco ha scritto: Il giorno gio, 27/08/2020 alle 00.28 +0200, Giuseppe Sacco ha scritto: Hello Christophe, Il giorno mer, 26/08/2020 alle 15.53 +0200, Christophe Leroy ha scritto: [...] If there is no warning, then the issue is something else, bad luck. Could you increase the loglevel and try again both with and without VMAP_STACK ? Maybe we'll get more information on where it stops. The problem is related to the CPU frequency changes. This is where the system stop: cpufreq get the CPU frequency limits and then start the default governor (performance) and then calls function cpufreq_gov_performance_limits() that never returns. Rebuilding after enabling pr_debug for cpufreq.c, I've got some more lines of output: cpufreq: setting new policy for CPU 0: 667000 - 867000 kHz cpufreq: new min and max freqs are 667000 - 867000 kHz cpufreq: governor switch cpufreq: cpufreq_init_governor: for CPU 0 cpufreq: cpufreq_start_governor: for CPU 0 cpufreq: target for CPU 0: 867000 kHz, relation 1, requested 867000 kHz cpufreq: __target_index: cpu: 0, oldfreq: 667000, new freq: 867000 cpufreq: notification 0 of frequency transition to 867000 kHz cpufreq: saving 133328 as reference value for loops_per_jiffy; freq is 667000 kHz no more lines are printed. I think this output only refers to the notification sent prior to the frequency change. I was thinking that selecting a governor that would run at 667mHz would probably skip the problem. I added cpufreq.default_governor=powersave to the command line parameters but it did not work: the selected governor was still performance and the system crashed. I kept following the thread and found that CPU frequency is changed in function pmu_set_cpu_speed() in file drivers/cpufreq/pmac32-cpufreq.c. As first thing, the function calls the macro preempt_disable() and this is where it stops. Sorry, I made a mistake. The real problem is down, on the same function, when it calls low_sleep_handler(). This is where the problem probably is. Great, you spotted the problem. I see what it is, it is in low_sleep_handler() in arch/powerpc/platforms/powermac/sleep.S All critical registers are saved on the stack. At restore, they are restore BEFORE re-enabling MMU (because they are needed for that). But when we have VMAP_STACK, the stack can hardly be accessed without the MMU enabled. tophys() doesn't work for virtual stack addresses. Therefore, the low_sleep_handler() has to be reworked for using an area in the linear mem instead of the stack. Christophe
Re: kernel since 5.6 do not boot anymore on Apple PowerBook
Il giorno gio, 27/08/2020 alle 09.46 +0200, Giuseppe Sacco ha scritto: > Il giorno gio, 27/08/2020 alle 00.28 +0200, Giuseppe Sacco ha scritto: > > Hello Christophe, > > > > Il giorno mer, 26/08/2020 alle 15.53 +0200, Christophe Leroy ha > > scritto: > > [...] > > > If there is no warning, then the issue is something else, bad luck. > > > > > > Could you increase the loglevel and try again both with and without > > > VMAP_STACK ? Maybe we'll get more information on where it stops. > > > > The problem is related to the CPU frequency changes. This is where the > > system stop: cpufreq get the CPU frequency limits and then start the > > default governor (performance) and then calls function > > cpufreq_gov_performance_limits() that never returns. > > > > Rebuilding after enabling pr_debug for cpufreq.c, I've got some more > > lines of output: > > > > cpufreq: setting new policy for CPU 0: 667000 - 867000 kHz > > cpufreq: new min and max freqs are 667000 - 867000 kHz > > cpufreq: governor switch > > cpufreq: cpufreq_init_governor: for CPU 0 > > cpufreq: cpufreq_start_governor: for CPU 0 > > cpufreq: target for CPU 0: 867000 kHz, relation 1, requested 867000 kHz > > cpufreq: __target_index: cpu: 0, oldfreq: 667000, new freq: 867000 > > cpufreq: notification 0 of frequency transition to 867000 kHz > > cpufreq: saving 133328 as reference value for loops_per_jiffy; freq is > > 667000 kHz > > > > no more lines are printed. I think this output only refers to the > > notification sent prior to the frequency change. > > > > I was thinking that selecting a governor that would run at 667mHz would > > probably skip the problem. I added cpufreq.default_governor=powersave > > to the command line parameters but it did not work: the selected > > governor was still performance and the system crashed. > > I kept following the thread and found that CPU frequency is changed in > function pmu_set_cpu_speed() in file drivers/cpufreq/pmac32-cpufreq.c. > As first thing, the function calls the macro preempt_disable() and this > is where it stops. Sorry, I made a mistake. The real problem is down, on the same function, when it calls low_sleep_handler(). This is where the problem probably is. Bye, Giuseppe
Re: kernel since 5.6 do not boot anymore on Apple PowerBook
Il giorno gio, 27/08/2020 alle 00.28 +0200, Giuseppe Sacco ha scritto: > Hello Christophe, > > Il giorno mer, 26/08/2020 alle 15.53 +0200, Christophe Leroy ha > scritto: > [...] > > If there is no warning, then the issue is something else, bad luck. > > > > Could you increase the loglevel and try again both with and without > > VMAP_STACK ? Maybe we'll get more information on where it stops. > > The problem is related to the CPU frequency changes. This is where the > system stop: cpufreq get the CPU frequency limits and then start the > default governor (performance) and then calls function > cpufreq_gov_performance_limits() that never returns. > > Rebuilding after enabling pr_debug for cpufreq.c, I've got some more > lines of output: > > cpufreq: setting new policy for CPU 0: 667000 - 867000 kHz > cpufreq: new min and max freqs are 667000 - 867000 kHz > cpufreq: governor switch > cpufreq: cpufreq_init_governor: for CPU 0 > cpufreq: cpufreq_start_governor: for CPU 0 > cpufreq: target for CPU 0: 867000 kHz, relation 1, requested 867000 kHz > cpufreq: __target_index: cpu: 0, oldfreq: 667000, new freq: 867000 > cpufreq: notification 0 of frequency transition to 867000 kHz > cpufreq: saving 133328 as reference value for loops_per_jiffy; freq is 667000 > kHz > > no more lines are printed. I think this output only refers to the > notification sent prior to the frequency change. > > I was thinking that selecting a governor that would run at 667mHz would > probably skip the problem. I added cpufreq.default_governor=powersave > to the command line parameters but it did not work: the selected > governor was still performance and the system crashed. I kept following the thread and found that CPU frequency is changed in function pmu_set_cpu_speed() in file drivers/cpufreq/pmac32-cpufreq.c. As first thing, the function calls the macro preempt_disable() and this is where it stops. Bye, Giuseppe
Re: kernel since 5.6 do not boot anymore on Apple PowerBook
Hello Christophe, Il giorno mer, 26/08/2020 alle 15.53 +0200, Christophe Leroy ha scritto: [...] > If there is no warning, then the issue is something else, bad luck. > > Could you increase the loglevel and try again both with and without > VMAP_STACK ? Maybe we'll get more information on where it stops. The problem is related to the CPU frequency changes. This is where the system stop: cpufreq get the CPU frequency limits and then start the default governor (performance) and then calls function cpufreq_gov_performance_limits() that never returns. Rebuilding after enabling pr_debug for cpufreq.c, I've got some more lines of output: cpufreq: setting new policy for CPU 0: 667000 - 867000 kHz cpufreq: new min and max freqs are 667000 - 867000 kHz cpufreq: governor switch cpufreq: cpufreq_init_governor: for CPU 0 cpufreq: cpufreq_start_governor: for CPU 0 cpufreq: target for CPU 0: 867000 kHz, relation 1, requested 867000 kHz cpufreq: __target_index: cpu: 0, oldfreq: 667000, new freq: 867000 cpufreq: notification 0 of frequency transition to 867000 kHz cpufreq: saving 133328 as reference value for loops_per_jiffy; freq is 667000 kHz no more lines are printed. I think this output only refers to the notification sent prior to the frequency change. I was thinking that selecting a governor that would run at 667mHz would probably skip the problem. I added cpufreq.default_governor=powersave to the command line parameters but it did not work: the selected governor was still performance and the system crashed. Bye, Giuseppe
Re: kernel since 5.6 do not boot anymore on Apple PowerBook
Hello Giuseppe, Le 24/08/2020 à 22:48, Giuseppe Sacco a écrit : Hello Christophe, Il giorno lun, 24/08/2020 alle 07.17 +0200, Christophe Leroy ha scritto: Hello Giuseppe, [...] The Oopses in the video are fixed in 5.9-rc2, see my response to your other mail. Right, I just updated from git and rebuilt the kernel whith CONFIG_VMAP_STACK not set and the machine boots correctly. So now we know that your kernel doesn't boot when CONFIG_VMAP_STACK is set. Can you remind the exact problem ? latest kernel with CONFIG_VMAP_STACK set stops after writing: pmac32_cpufreq: registering PowerMac CPU frequency driver pmac32_cpufreq: Low: 667 MHz, High: 867 Mhz, Boot: 667 MHz One common problem with CONFIG_VMAP_STACK is when some drivers are invalidly using buffers in stack for DMA. Couldn't try with CONFIG_DEBUG_VIRTUAL (without CONFIG_VMAP_STACK) and see if it triggers some warnings ? I've just tried: it boots without any special warning. What should I look for? This is an excerpt of dmesg output about the line it would otherwise stop: If there is no warning, then the issue is something else, bad luck. Could you increase the loglevel and try again both with and without VMAP_STACK ? Maybe we'll get more information on where it stops. Christophe [...] [6.566984] PowerMac i2c bus pmu 2 registered [6.574879] PowerMac i2c bus pmu 1 registered [6.582634] PowerMac i2c bus mac-io 0 registered [6.590323] i2c i2c-2: No i2c address for /pci@f200/mac-io@17/i2c@18000/i2c-modem [6.598290] PowerMac i2c bus uni-n 1 registered [6.606196] i2c i2c-3: i2c-powermac: modalias failure on /uni-n@f800/i2c@f8001000/cereal@1c0 [6.614320] PowerMac i2c bus uni-n 0 registered [6.622501] pmac32_cpufreq: Registering PowerMac CPU frequency driver [6.630580] pmac32_cpufreq: Low: 667 Mhz, High: 867 Mhz, Boot: 667 Mhz [6.639518] ledtrig-cpu: registered to indicate activity on CPUs [6.647894] NET: Registered protocol family 10 [6.656492] Segment Routing with IPv6 [6.664490] mip6: Mobile IPv6 [6.672337] NET: Registered protocol family 17 [6.680213] mpls_gso: MPLS GSO support [...] Bye, Giuseppe
Re: kernel since 5.6 do not boot anymore on Apple PowerBook
Hello Giuseppe, Le 24/08/2020 à 22:48, Giuseppe Sacco a écrit : Hello Christophe, Il giorno lun, 24/08/2020 alle 07.17 +0200, Christophe Leroy ha scritto: Hello Giuseppe, [...] The Oopses in the video are fixed in 5.9-rc2, see my response to your other mail. Right, I just updated from git and rebuilt the kernel whith CONFIG_VMAP_STACK not set and the machine boots correctly. So now we know that your kernel doesn't boot when CONFIG_VMAP_STACK is set. Can you remind the exact problem ? latest kernel with CONFIG_VMAP_STACK set stops after writing: pmac32_cpufreq: registering PowerMac CPU frequency driver pmac32_cpufreq: Low: 667 MHz, High: 867 Mhz, Boot: 667 MHz One common problem with CONFIG_VMAP_STACK is when some drivers are invalidly using buffers in stack for DMA. Couldn't try with CONFIG_DEBUG_VIRTUAL (without CONFIG_VMAP_STACK) and see if it triggers some warnings ? I've just tried: it boots without any special warning. What should I look for? This is an excerpt of dmesg output about the line it would otherwise stop: If there is no warning, then the issue is something else, bad luck. Could you increase the loglevel and try again both with and without VMAP_STACK ? Maybe we'll get more information on where it stops. Christophe [...] [ 6.566984] PowerMac i2c bus pmu 2 registered [ 6.574879] PowerMac i2c bus pmu 1 registered [ 6.582634] PowerMac i2c bus mac-io 0 registered [ 6.590323] i2c i2c-2: No i2c address for /pci@f200/mac-io@17/i2c@18000/i2c-modem [ 6.598290] PowerMac i2c bus uni-n 1 registered [ 6.606196] i2c i2c-3: i2c-powermac: modalias failure on /uni-n@f800/i2c@f8001000/cereal@1c0 [ 6.614320] PowerMac i2c bus uni-n 0 registered [ 6.622501] pmac32_cpufreq: Registering PowerMac CPU frequency driver [ 6.630580] pmac32_cpufreq: Low: 667 Mhz, High: 867 Mhz, Boot: 667 Mhz [ 6.639518] ledtrig-cpu: registered to indicate activity on CPUs [ 6.647894] NET: Registered protocol family 10 [ 6.656492] Segment Routing with IPv6 [ 6.664490] mip6: Mobile IPv6 [ 6.672337] NET: Registered protocol family 17 [ 6.680213] mpls_gso: MPLS GSO support [...] Bye, Giuseppe
Re: kernel since 5.6 do not boot anymore on Apple PowerBook
Hello Christophe, Il giorno lun, 24/08/2020 alle 07.17 +0200, Christophe Leroy ha scritto: > Hello Giuseppe, [...] > The Oopses in the video are fixed in 5.9-rc2, see my response to your > other mail. Right, I just updated from git and rebuilt the kernel whith CONFIG_VMAP_STACK not set and the machine boots correctly. > So now we know that your kernel doesn't boot when CONFIG_VMAP_STACK is set. > Can you remind the exact problem ? latest kernel with CONFIG_VMAP_STACK set stops after writing: pmac32_cpufreq: registering PowerMac CPU frequency driver pmac32_cpufreq: Low: 667 MHz, High: 867 Mhz, Boot: 667 MHz > One common problem with CONFIG_VMAP_STACK is when some drivers are > invalidly using buffers in stack for DMA. > > Couldn't try with CONFIG_DEBUG_VIRTUAL (without CONFIG_VMAP_STACK) and > see if it triggers some warnings ? I've just tried: it boots without any special warning. What should I look for? This is an excerpt of dmesg output about the line it would otherwise stop: [...] [6.566984] PowerMac i2c bus pmu 2 registered [6.574879] PowerMac i2c bus pmu 1 registered [6.582634] PowerMac i2c bus mac-io 0 registered [6.590323] i2c i2c-2: No i2c address for /pci@f200/mac-io@17/i2c@18000/i2c-modem [6.598290] PowerMac i2c bus uni-n 1 registered [6.606196] i2c i2c-3: i2c-powermac: modalias failure on /uni-n@f800/i2c@f8001000/cereal@1c0 [6.614320] PowerMac i2c bus uni-n 0 registered [6.622501] pmac32_cpufreq: Registering PowerMac CPU frequency driver [6.630580] pmac32_cpufreq: Low: 667 Mhz, High: 867 Mhz, Boot: 667 Mhz [6.639518] ledtrig-cpu: registered to indicate activity on CPUs [6.647894] NET: Registered protocol family 10 [6.656492] Segment Routing with IPv6 [6.664490] mip6: Mobile IPv6 [6.672337] NET: Registered protocol family 17 [6.680213] mpls_gso: MPLS GSO support [...] Bye, Giuseppe
Re: kernel since 5.6 do not boot anymore on Apple PowerBook
Hello Giuseppe, Le 22/08/2020 à 10:28, Giuseppe Sacco a écrit : Hello Christophe, Il giorno ven, 21/08/2020 alle 16.03 +0200, Christophe Leroy ha scritto: [...] You also said in a previous mail that your original issue also happens when CONFIG_VMAP_STACK is not selected. The above bug being linked to CONFIG_VMAP_STACK, maybe it would be easier to bisect with CONFIG_VMAP_STACK unselected. I was wrong. Disabling CONFIG_VMAP_STACK led me to all "good" compile and bisect ended without finding the culprit commit. So, I started from scratch: I rebuilt HEAD and found that it does show the original problem I am facing, then I rebuilt it without CONFIG_VMAP_STACK and found that it does pass (fix?) the problem, since kernel continue booting, but then it stops with three Oops related to command systemd-udevd. You may find a video that displays the complete boot, vmlinux, config, and system.map files here: The Oopses in the video are fixed in 5.9-rc2, see my response to your other mail. So now we know that your kernel doesn't boot when CONFIG_VMAP_STACK is set. Can you remind the exact problem ? One common problem with CONFIG_VMAP_STACK is when some drivers are invalidly using buffers in stack for DMA. Couldn't try with CONFIG_DEBUG_VIRTUAL (without CONFIG_VMAP_STACK) and see if it triggers some warnings ? Thanks Christophe
Re: kernel since 5.6 do not boot anymore on Apple PowerBook
Hello Christophe, Il giorno ven, 21/08/2020 alle 16.03 +0200, Christophe Leroy ha scritto: [...] > Thanks. > > The Oops in the video shows that the issue is at 0x1bcac and msr > value > shows that Instruction MMU is disabled. So this corresponds to > address > 0xc001bcac. In the vmlinux you sent me this address is in > power_save_ppc32_restore() > > This issue is fixed by > https://patchwork.ozlabs.org/project/linuxppc-dev/patch/7bce32ccbab3ba3e3e0f27da6961bf6313df97ed.1581663140.git.christophe.le...@c-s.fr/ > > > You also said in a previous mail that your original issue also > happens > when CONFIG_VMAP_STACK is not selected. The above bug being linked > to > CONFIG_VMAP_STACK, maybe it would be easier to bisect with > CONFIG_VMAP_STACK unselected. I was wrong. Disabling CONFIG_VMAP_STACK led me to all "good" compile and bisect ended without finding the culprit commit. So, I started from scratch: I rebuilt HEAD and found that it does show the original problem I am facing, then I rebuilt it without CONFIG_VMAP_STACK and found that it does pass (fix?) the problem, since kernel continue booting, but then it stops with three Oops related to command systemd-udevd. You may find a video that displays the complete boot, vmlinux, config, and system.map files here: https://eppesuigoccas.homedns.org/~giuseppe/powerpc32/config-5.9.0-rc1+ https://eppesuigoccas.homedns.org/~giuseppe/powerpc32/System.map https://eppesuigoccas.homedns.org/~giuseppe/powerpc32/VID_20200822_095621.mp4 https://eppesuigoccas.homedns.org/~giuseppe/powerpc32/vmlinux.strip.gz Bye, Giuseppe
Re: kernel since 5.6 do not boot anymore on Apple PowerBook
Hello, Il giorno ven, 21/08/2020 alle 16.03 +0200, Christophe Leroy ha scritto: [...] > Thanks. > > The Oops in the video shows that the issue is at 0x1bcac and msr > value > shows that Instruction MMU is disabled. So this corresponds to > address > 0xc001bcac. In the vmlinux you sent me this address is in > power_save_ppc32_restore() > > This issue is fixed by > https://patchwork.ozlabs.org/project/linuxppc-dev/patch/7bce32ccbab3ba3e3e0f27da6961bf6313df97ed.1581663140.git.christophe.le...@c-s.fr/ > > > You also said in a previous mail that your original issue also > happens > when CONFIG_VMAP_STACK is not selected. The above bug being linked > to > CONFIG_VMAP_STACK, maybe it would be easier to bisect with > CONFIG_VMAP_STACK unselected. I rebuilt the same kernel that I would otherwise have skipped because of the other bug, and it worked, indeed. I will continue my "bisect quest" unselecting VMAP_STACK in order to speed up the bisecting. Thank you very much, Giuseppe
Re: kernel since 5.6 do not boot anymore on Apple PowerBook
Le 21/08/2020 à 15:50, Giuseppe Sacco a écrit : Il giorno ven, 21/08/2020 alle 15.29 +0200, Christophe Leroy ha scritto: [...] Maybe the easiest would be first to locate this issue. Can you send me the vmlinux and the .config matching the Oops in the video ? And also the output of git bisect log ? Here it is. Not really the one on the video, but the last "skipped" for the same problem. https://eppesuigoccas.homedns.org/~giuseppe/christophe.tar.bz2 Thanks. The Oops in the video shows that the issue is at 0x1bcac and msr value shows that Instruction MMU is disabled. So this corresponds to address 0xc001bcac. In the vmlinux you sent me this address is in power_save_ppc32_restore() This issue is fixed by https://patchwork.ozlabs.org/project/linuxppc-dev/patch/7bce32ccbab3ba3e3e0f27da6961bf6313df97ed.1581663140.git.christophe.le...@c-s.fr/ You also said in a previous mail that your original issue also happens when CONFIG_VMAP_STACK is not selected. The above bug being linked to CONFIG_VMAP_STACK, maybe it would be easier to bisect with CONFIG_VMAP_STACK unselected. Christophe
Re: kernel since 5.6 do not boot anymore on Apple PowerBook
Hi again, Le 21/08/2020 à 10:22, Giuseppe Sacco a écrit : Hello Cristophe, Il giorno ven, 21/08/2020 alle 08.55 +0200, Christophe Leroy ha scritto: Hi Giuseppe, Le 08/07/2020 à 20:44, Christophe Leroy a écrit : Le 08/07/2020 à 19:36, Giuseppe Sacco a écrit : Hi Cristophe, Il giorno mer, 08/07/2020 alle 19.09 +0200, Christophe Leroy ha scritto: [...] What's the result with: LANG=C make ARCH=powerpc CROSS_COMPILE=powerpc-linux- vmlinux $ LANG=C make ARCH=powerpc CROSS_COMPILE=powerpc-linux- vmlinux CALLscripts/checksyscalls.sh CALLscripts/atomic/check-atomics.sh CHK include/generated/compile.h CC kernel/module.o kernel/module.c: In function 'do_init_module': kernel/module.c:3593:2: error: implicit declaration of function 'module_enable_ro'; did you mean 'module_enable_x'? [-Werror=implicit- function-declaration] 3593 | module_enable_ro(mod, true); | ^~~~ | module_enable_x cc1: some warnings being treated as errors make[1]: *** [scripts/Makefile.build:267: kernel/module.o] Error 1 make: *** [Makefile:1735: kernel] Error 2 So, should I 'git bisect skip'? Ah yes, I had the exact same problem last time I bisected. So yes do 'git bisect skip'. You'll probably hit this problem half a dozen of times, but at the end you should get a usefull bisect anyway. Were you able to progress ? Very slowly. I am still working on it, currently at recompile #276. git-bisect states that I have still about 700 commits to check, but the real problem is that more than 60% of built kernels crash even before displaying the cpu_freq message (probably another long lasting bug hides the one I am looking for). All these skipped kernels make bisecting very very slow. A short video about the problem I face when I skip the build is here: https://eppesuigoccas.homedns.org/~giuseppe/bug%20avvio%20powerbook%20g4.mp4 Maybe the easiest would be first to locate this issue. Can you send me the vmlinux and the .config matching the Oops in the video ? And also the output of git bisect log ? Thanks Christophe
Re: kernel since 5.6 do not boot anymore on Apple PowerBook
Hello Cristophe, Il giorno ven, 21/08/2020 alle 08.55 +0200, Christophe Leroy ha scritto: > Hi Giuseppe, > > Le 08/07/2020 à 20:44, Christophe Leroy a écrit : > > > > Le 08/07/2020 à 19:36, Giuseppe Sacco a écrit : > > > Hi Cristophe, > > > > > > Il giorno mer, 08/07/2020 alle 19.09 +0200, Christophe Leroy ha > > > scritto: [...] > > > > What's the result with: > > > > > > > > LANG=C make ARCH=powerpc CROSS_COMPILE=powerpc-linux- vmlinux > > > > > > $ LANG=C make ARCH=powerpc CROSS_COMPILE=powerpc-linux- vmlinux > > >CALLscripts/checksyscalls.sh > > >CALLscripts/atomic/check-atomics.sh > > >CHK include/generated/compile.h > > >CC kernel/module.o > > > kernel/module.c: In function 'do_init_module': > > > kernel/module.c:3593:2: error: implicit declaration of function > > > 'module_enable_ro'; did you mean 'module_enable_x'? [-Werror=implicit- > > > function-declaration] > > > 3593 | module_enable_ro(mod, true); > > >| ^~~~ > > >| module_enable_x > > > cc1: some warnings being treated as errors > > > make[1]: *** [scripts/Makefile.build:267: kernel/module.o] Error 1 > > > make: *** [Makefile:1735: kernel] Error 2 > > > > > > So, should I 'git bisect skip'? > > > > Ah yes, I had the exact same problem last time I bisected. > > > > So yes do 'git bisect skip'. You'll probably hit this problem half a > > dozen of times, but at the end you should get a usefull bisect anyway. > > > > Were you able to progress ? Very slowly. I am still working on it, currently at recompile #276. git-bisect states that I have still about 700 commits to check, but the real problem is that more than 60% of built kernels crash even before displaying the cpu_freq message (probably another long lasting bug hides the one I am looking for). All these skipped kernels make bisecting very very slow. A short video about the problem I face when I skip the build is here: https://eppesuigoccas.homedns.org/~giuseppe/bug%20avvio%20powerbook%20g4.mp4 Bye, Giuseppe
Re: kernel since 5.6 do not boot anymore on Apple PowerBook
Hi Giuseppe, Le 08/07/2020 à 20:44, Christophe Leroy a écrit : Le 08/07/2020 à 19:36, Giuseppe Sacco a écrit : Hi Cristophe, Il giorno mer, 08/07/2020 alle 19.09 +0200, Christophe Leroy ha scritto: Hi Le 08/07/2020 à 19:00, Giuseppe Sacco a écrit : Hello, while trying to debug a problem using git bisect, I am now at a point where I cannot build the kernel at all. This is the error message I get: $ LANG=C make ARCH=powerpc \ CROSS_COMPILE=powerpc-linux- \ CONFIG_MODULE_COMPRESS_GZIP=true \ INSTALL_MOD_STRIP=1 CONFIG_MODULE_COMPRESS=1 \ -j4 INSTALL_MOD_PATH=$BOOT INSTALL_PATH=$BOOT \ CONFIG_DEBUG_INFO_COMPRESSED=1 \ install modules_install make[2]: *** No rule to make target 'vmlinux', needed by Surprising. Did you make any change to Makefiles ? No Are you in the middle of a bisect ? If so, if the previous builds worked, I'd do 'git bisect skip' Yes, the previous one worked. What's the result with: LANG=C make ARCH=powerpc CROSS_COMPILE=powerpc-linux- vmlinux $ LANG=C make ARCH=powerpc CROSS_COMPILE=powerpc-linux- vmlinux CALL scripts/checksyscalls.sh CALL scripts/atomic/check-atomics.sh CHK include/generated/compile.h CC kernel/module.o kernel/module.c: In function 'do_init_module': kernel/module.c:3593:2: error: implicit declaration of function 'module_enable_ro'; did you mean 'module_enable_x'? [-Werror=implicit- function-declaration] 3593 | module_enable_ro(mod, true); | ^~~~ | module_enable_x cc1: some warnings being treated as errors make[1]: *** [scripts/Makefile.build:267: kernel/module.o] Error 1 make: *** [Makefile:1735: kernel] Error 2 So, should I 'git bisect skip'? Ah yes, I had the exact same problem last time I bisected. So yes do 'git bisect skip'. You'll probably hit this problem half a dozen of times, but at the end you should get a usefull bisect anyway. Were you able to progress ? Christophe
Re: kernel since 5.6 do not boot anymore on Apple PowerBook
Le 08/07/2020 à 19:36, Giuseppe Sacco a écrit : Hi Cristophe, Il giorno mer, 08/07/2020 alle 19.09 +0200, Christophe Leroy ha scritto: Hi Le 08/07/2020 à 19:00, Giuseppe Sacco a écrit : Hello, while trying to debug a problem using git bisect, I am now at a point where I cannot build the kernel at all. This is the error message I get: $ LANG=C make ARCH=powerpc \ CROSS_COMPILE=powerpc-linux- \ CONFIG_MODULE_COMPRESS_GZIP=true \ INSTALL_MOD_STRIP=1 CONFIG_MODULE_COMPRESS=1 \ -j4 INSTALL_MOD_PATH=$BOOT INSTALL_PATH=$BOOT \ CONFIG_DEBUG_INFO_COMPRESSED=1 \ install modules_install make[2]: *** No rule to make target 'vmlinux', needed by Surprising. Did you make any change to Makefiles ? No Are you in the middle of a bisect ? If so, if the previous builds worked, I'd do 'git bisect skip' Yes, the previous one worked. What's the result with: LANG=C make ARCH=powerpc CROSS_COMPILE=powerpc-linux- vmlinux $ LANG=C make ARCH=powerpc CROSS_COMPILE=powerpc-linux- vmlinux CALLscripts/checksyscalls.sh CALLscripts/atomic/check-atomics.sh CHK include/generated/compile.h CC kernel/module.o kernel/module.c: In function 'do_init_module': kernel/module.c:3593:2: error: implicit declaration of function 'module_enable_ro'; did you mean 'module_enable_x'? [-Werror=implicit- function-declaration] 3593 | module_enable_ro(mod, true); | ^~~~ | module_enable_x cc1: some warnings being treated as errors make[1]: *** [scripts/Makefile.build:267: kernel/module.o] Error 1 make: *** [Makefile:1735: kernel] Error 2 So, should I 'git bisect skip'? Ah yes, I had the exact same problem last time I bisected. So yes do 'git bisect skip'. You'll probably hit this problem half a dozen of times, but at the end you should get a usefull bisect anyway. Christophe
Re: kernel since 5.6 do not boot anymore on Apple PowerBook
Hi Cristophe, Il giorno mer, 08/07/2020 alle 19.09 +0200, Christophe Leroy ha scritto: > Hi > > Le 08/07/2020 à 19:00, Giuseppe Sacco a écrit : > > Hello, > > while trying to debug a problem using git bisect, I am now at a point > > where I cannot build the kernel at all. This is the error message I > > get: > > > > $ LANG=C make ARCH=powerpc \ > > CROSS_COMPILE=powerpc-linux- \ > > CONFIG_MODULE_COMPRESS_GZIP=true \ > > INSTALL_MOD_STRIP=1 CONFIG_MODULE_COMPRESS=1 \ > > -j4 INSTALL_MOD_PATH=$BOOT INSTALL_PATH=$BOOT \ > > CONFIG_DEBUG_INFO_COMPRESSED=1 \ > > install modules_install > > make[2]: *** No rule to make target 'vmlinux', needed by > > Surprising. > > Did you make any change to Makefiles ? No > Are you in the middle of a bisect ? If so, if the previous builds > worked, I'd do 'git bisect skip' Yes, the previous one worked. > What's the result with: > > LANG=C make ARCH=powerpc CROSS_COMPILE=powerpc-linux- vmlinux $ LANG=C make ARCH=powerpc CROSS_COMPILE=powerpc-linux- vmlinux CALLscripts/checksyscalls.sh CALLscripts/atomic/check-atomics.sh CHK include/generated/compile.h CC kernel/module.o kernel/module.c: In function 'do_init_module': kernel/module.c:3593:2: error: implicit declaration of function 'module_enable_ro'; did you mean 'module_enable_x'? [-Werror=implicit- function-declaration] 3593 | module_enable_ro(mod, true); | ^~~~ | module_enable_x cc1: some warnings being treated as errors make[1]: *** [scripts/Makefile.build:267: kernel/module.o] Error 1 make: *** [Makefile:1735: kernel] Error 2 So, should I 'git bisect skip'? Thank you, Giuseppe
Re: kernel since 5.6 do not boot anymore on Apple PowerBook
Hi Le 08/07/2020 à 19:00, Giuseppe Sacco a écrit : Hello, while trying to debug a problem using git bisect, I am now at a point where I cannot build the kernel at all. This is the error message I get: $ LANG=C make ARCH=powerpc \ CROSS_COMPILE=powerpc-linux- \ CONFIG_MODULE_COMPRESS_GZIP=true \ INSTALL_MOD_STRIP=1 CONFIG_MODULE_COMPRESS=1 \ -j4 INSTALL_MOD_PATH=$BOOT INSTALL_PATH=$BOOT \ CONFIG_DEBUG_INFO_COMPRESSED=1 \ install modules_install make[2]: *** No rule to make target 'vmlinux', needed by Surprising. Did you make any change to Makefiles ? Are you in the middle of a bisect ? If so, if the previous builds worked, I'd do 'git bisect skip' What's the result with: LANG=C make ARCH=powerpc CROSS_COMPILE=powerpc-linux- vmlinux Christophe 'arch/powerpc/boot/zImage.pmac'. Stop. make[1]: *** [arch/powerpc/Makefile:407: install] Error 2 make: *** [Makefile:328: __build_one_by_one] Error 2 How can I continue? Thank you, Giuseppe
Re: kernel since 5.6 do not boot anymore on Apple PowerBook
Hello, while trying to debug a problem using git bisect, I am now at a point where I cannot build the kernel at all. This is the error message I get: $ LANG=C make ARCH=powerpc \ CROSS_COMPILE=powerpc-linux- \ CONFIG_MODULE_COMPRESS_GZIP=true \ INSTALL_MOD_STRIP=1 CONFIG_MODULE_COMPRESS=1 \ -j4 INSTALL_MOD_PATH=$BOOT INSTALL_PATH=$BOOT \ CONFIG_DEBUG_INFO_COMPRESSED=1 \ install modules_install make[2]: *** No rule to make target 'vmlinux', needed by 'arch/powerpc/boot/zImage.pmac'. Stop. make[1]: *** [arch/powerpc/Makefile:407: install] Error 2 make: *** [Makefile:328: __build_one_by_one] Error 2 How can I continue? Thank you, Giuseppe
Re: kernel since 5.6 do not boot anymore on Apple PowerBook
Salut Cristophe, Il giorno mer, 08/07/2020 alle 10.38 +0200, Christophe Leroy ha scritto: > Hi Giuseppe, > > Le 08/07/2020 à 09:59, Giuseppe Sacco a écrit : > > Hello Cristophe, > > > > Il giorno mar, 07/07/2020 alle 17.34 +0200, Giuseppe Sacco ha > > scritto: [...] > > > > And can you try without CONFIG_VMAP_STACK on 5.6.19 > > > > > > Sure, I'll let you know. > > > > No, this change did not make the kernel boot. I only changed the > > option > > you proposed: > > > > $ grep VMAP .config > > CONFIG_HAVE_ARCH_VMAP_STACK=y > > # CONFIG_VMAP_STACK is not set > > > > Ok, at least it is not that. I wanted to be sure because this is a > huge > change added recently that is selected by default. > > I tried on QEMU with your config, it boots properly. > > So I think the only way now will be to bisect. Hope you were able to > setup a cross compiler on a speedy machine. thank you for your time; indeed I may now crosscompile the kernel in 15 minutes instead of about 390... I hope to have a final result in a couple of days. Bye, Giuseppe
Re: kernel since 5.6 do not boot anymore on Apple PowerBook
Hi Giuseppe, Le 08/07/2020 à 09:59, Giuseppe Sacco a écrit : Hello Cristophe, Il giorno mar, 07/07/2020 alle 17.34 +0200, Giuseppe Sacco ha scritto: Il giorno mar, 07/07/2020 alle 16.52 +0200, Christophe Leroy ha scritto: Le 07/07/2020 à 16:03, Giuseppe Sacco a écrit : Hello Cristophe, Can you tell which defconfig you use or provide your .config You may get the standard one from debian or a reduced one that I made on purpose. The latter is here: https://eppesuigoccas.homedns.org/~giuseppe/config-5.4.50.gz (boot) https://eppesuigoccas.homedns.org/~giuseppe/config-5.6.19.gz (no boot) Thanks Can you provide the complete output when it works, so that I can see what is after the place it stops when it fails. Here it is: https://eppesuigoccas.homedns.org/~giuseppe/dmesg-5.4.40-minimo.gz And can you try without CONFIG_VMAP_STACK on 5.6.19 Sure, I'll let you know. No, this change did not make the kernel boot. I only changed the option you proposed: $ grep VMAP .config CONFIG_HAVE_ARCH_VMAP_STACK=y # CONFIG_VMAP_STACK is not set Ok, at least it is not that. I wanted to be sure because this is a huge change added recently that is selected by default. I tried on QEMU with your config, it boots properly. So I think the only way now will be to bisect. Hope you were able to setup a cross compiler on a speedy machine. Christophe
Re: kernel since 5.6 do not boot anymore on Apple PowerBook
Hello Cristophe, Il giorno mar, 07/07/2020 alle 17.34 +0200, Giuseppe Sacco ha scritto: > Il giorno mar, 07/07/2020 alle 16.52 +0200, Christophe Leroy ha > scritto: > > Le 07/07/2020 à 16:03, Giuseppe Sacco a écrit : > > > Hello Cristophe, > > > > > > > Can you tell which defconfig you use or provide your .config > > > > > > You may get the standard one from debian or a reduced one that I > > > made on purpose. The latter is here: > > > https://eppesuigoccas.homedns.org/~giuseppe/config-5.4.50.gz > > > > > > (boot) > > > https://eppesuigoccas.homedns.org/~giuseppe/config-5.6.19.gz > > > > > > (no boot) > > > > Thanks > > > > Can you provide the complete output when it works, so that I can > > see > > what is after the place it stops when it fails. > > Here it is: > https://eppesuigoccas.homedns.org/~giuseppe/dmesg-5.4.40-minimo.gz > > > > And can you try without CONFIG_VMAP_STACK on 5.6.19 > > Sure, I'll let you know. No, this change did not make the kernel boot. I only changed the option you proposed: $ grep VMAP .config CONFIG_HAVE_ARCH_VMAP_STACK=y # CONFIG_VMAP_STACK is not set Thanks, Giuseppe
Re: kernel since 5.6 do not boot anymore on Apple PowerBook
Il giorno mar, 07/07/2020 alle 16.52 +0200, Christophe Leroy ha scritto: > > Le 07/07/2020 à 16:03, Giuseppe Sacco a écrit : > > Hello Cristophe, > > > > > Can you tell which defconfig you use or provide your .config > > > > You may get the standard one from debian or a reduced one that I > > made on purpose. The latter is here: > > https://eppesuigoccas.homedns.org/~giuseppe/config-5.4.50.gz > > (boot) > > https://eppesuigoccas.homedns.org/~giuseppe/config-5.6.19.gz > > (no boot) > > Thanks > > Can you provide the complete output when it works, so that I can see > what is after the place it stops when it fails. Here it is: https://eppesuigoccas.homedns.org/~giuseppe/dmesg-5.4.40-minimo.gz > And can you try without CONFIG_VMAP_STACK on 5.6.19 Sure, I'll let you know. Thank you, Giuseppe
Re: kernel since 5.6 do not boot anymore on Apple PowerBook
Le 07/07/2020 à 16:03, Giuseppe Sacco a écrit : Hello Cristophe, Il giorno mar, 07/07/2020 alle 15.35 +0200, Christophe Leroy ha scritto: Hi, Le 07/07/2020 à 14:58, Giuseppe Sacco a écrit : [...] So, I am looking for help. How may I better investigate the problem? How may I crossbuild the kernel on a faster machine (amd64) in order to try git-bisect without waiting ages? To crossbuild, use cross tools at https://mirrors.edge.kernel.org/pub/tools/crosstool/ thank you, I'll have a look at it. Can you tell which defconfig you use or provide your .config You may get the standard one from debian or a reduced one that I made on purpose. The latter is here: https://eppesuigoccas.homedns.org/~giuseppe/config-5.4.50.gz (boot) https://eppesuigoccas.homedns.org/~giuseppe/config-5.6.19.gz (no boot) Thanks Can you provide the complete output when it works, so that I can see what is after the place it stops when it fails. And can you try without CONFIG_VMAP_STACK on 5.6.19 Christophe
Re: kernel since 5.6 do not boot anymore on Apple PowerBook
Hello Cristophe, Il giorno mar, 07/07/2020 alle 15.35 +0200, Christophe Leroy ha scritto: > Hi, > Le 07/07/2020 à 14:58, Giuseppe Sacco a écrit : [...] > > So, I am looking for help. How may I better investigate the > > problem? > > How may I crossbuild the kernel on a faster machine (amd64) in > > order to > > try git-bisect without waiting ages? > > To crossbuild, use cross tools at > https://mirrors.edge.kernel.org/pub/tools/crosstool/ thank you, I'll have a look at it. > Can you tell which defconfig you use or provide your .config You may get the standard one from debian or a reduced one that I made on purpose. The latter is here: https://eppesuigoccas.homedns.org/~giuseppe/config-5.4.50.gz (boot) https://eppesuigoccas.homedns.org/~giuseppe/config-5.6.19.gz (no boot) Bye, Giuseppe
Re: kernel since 5.6 do not boot anymore on Apple PowerBook
Hi, Le 07/07/2020 à 14:58, Giuseppe Sacco a écrit : Hello, I am testing linux on a quite old hardware, an Apple PowerBook G4 "Titanium IV". I used to run Debian on this machine, so I upgraded the old operating system to the latest software and found that new kernels do not boot. I rebuilt natively (on the powerbook) a few kernels and found that 5.4.50 still works fine, while 5.6.19 and 5.7.7 stop after printing: pmac32_cpufreq: registering PowerMac CPU frequency driver pmac32_cpufreq: Low: 667 MHz, High: 867 Mhz, Boot: 667 MHz no more output is printed. I waited 5 minutes and then powered off the machine using the power button. Rebooting with old kernel works. So, I am looking for help. How may I better investigate the problem? How may I crossbuild the kernel on a faster machine (amd64) in order to try git-bisect without waiting ages? To crossbuild, use cross tools at https://mirrors.edge.kernel.org/pub/tools/crosstool/ Can you tell which defconfig you use or provide your .config Christophe
kernel since 5.6 do not boot anymore on Apple PowerBook
Hello, I am testing linux on a quite old hardware, an Apple PowerBook G4 "Titanium IV". I used to run Debian on this machine, so I upgraded the old operating system to the latest software and found that new kernels do not boot. I rebuilt natively (on the powerbook) a few kernels and found that 5.4.50 still works fine, while 5.6.19 and 5.7.7 stop after printing: pmac32_cpufreq: registering PowerMac CPU frequency driver pmac32_cpufreq: Low: 667 MHz, High: 867 Mhz, Boot: 667 MHz no more output is printed. I waited 5 minutes and then powered off the machine using the power button. Rebooting with old kernel works. So, I am looking for help. How may I better investigate the problem? How may I crossbuild the kernel on a faster machine (amd64) in order to try git-bisect without waiting ages? Thank you, Giuseppe
