please have a look at the log messages of classification example. am seeing huge_size as zero, is it correct ? in "/proc/meminfo" Hugepagesize is 2048 kB.
root@ubuntu-15-10:/home/linaro/linaro/odp/example/classifier# ./odp_classifier -i eno1 -m 0 -p "ODP_PMR_SIP_ADDR:192.168.10.11:FFFFFFFF:queue1" -p "ODP_PMR_SIP_ADDR:10.130.69.0:000000FF:queue2" -p "ODP_PMR_SIP_ADDR:10.130.68.0:FFFFFE00:queue3" odp_shm_reserve : page_sz 4096 alloc_size 34384 *huge_sze 0* On Wed, May 11, 2016 at 1:59 PM, Maxim Uvarov <[email protected]> wrote: > > > On 11 May 2016 at 09:04, nousi <[email protected]> wrote: > >> Hi Maxim, >> >> To return 2MB we need to change the logic in huge_page_size() >> (odP-system_info.c)function as below. >> Do not know about old logic. >> >> I do not see huge page allocation error if I change the logic as below. >> >> diff --git a/platform/linux-generic/odp_system_info.c >> b/platform/linux-generic/odp_system_info.c >> index 0f1f3c7..2173763 100644 >> --- a/platform/linux-generic/odp_system_info.c >> +++ b/platform/linux-generic/odp_system_info.c >> @@ -95,7 +95,7 @@ static int huge_page_size(void) >> if (sscanf(dirent->d_name, "hugepages-%i", &temp) != 1) >> continue; >> >> - if (temp > size) >> + if (temp < size) >> size = temp; >> } >> >> >> > > No, that is not correct. You request for HP and if huge page is less then > requested size you limit requested size. > Problem is solved with using default system huge pages instead of first > found in sysfs. > Try this patch: > http://patches.opendataplane.org/patch/5934/ > > Maxim. > > >> On Tue, May 10, 2016 at 6:38 PM, Maxim Uvarov <[email protected]> >> wrote: >> >>> On 05/10/16 15:53, nousi wrote: >>> >>>> value is zero in all the huge page parameters. >>>> >>>> >>> That is not the problem. Problem is that we have call: >>> >>> /** >>> * Huge page size in bytes >>> * >>> * @return Huge page size in bytes >>> */ >>> uint64_t odp_sys_huge_page_size(void); >>> >>> which worked well for only one huge page sizes. >>> >>> >>> platform/linux-generic/odp_system_info.c >>> static int huge_page_size(void) >>> { >>> if (sscanf(dirent->d_name, "hugepages-%i", &temp) != 1) >>> continue; >>> >>> >>> So because 1 GB pages in /proc lists first now then >>> odp_sys_huge_page_size() returns 1 GB, not 2 MB pages. >>> >>> In general we need to think how to fix it cleanly. Probably modify api >>> of odp_sys_huge_page_size() to return >>> all available huge page sizes. >>> >>> So it looks like a bug, thank you for taking attention about it. >>> >>> Maxim. >>> >>> >>> >>> root@ubuntu-15-10:/sys/kernel/mm/hugepages/hugepages-1048576kB# ls -l >>>> total 0 >>>> -r--r--r-- 1 root root 4096 May 10 18:15 free_hugepages >>>> -rw-r--r-- 1 root root 4096 May 10 18:15 nr_hugepages >>>> -rw-r--r-- 1 root root 4096 May 10 18:15 nr_hugepages_mempolicy >>>> -rw-r--r-- 1 root root 4096 May 10 18:15 nr_overcommit_hugepages >>>> -r--r--r-- 1 root root 4096 May 10 18:15 resv_hugepages >>>> -r--r--r-- 1 root root 4096 May 10 18:15 surplus_hugepages >>>> root@ubuntu-15-10:/sys/kernel/mm/hugepages/hugepages-1048576kB# cat >>>> free_hugepages >>>> 0 >>>> root@ubuntu-15-10:/sys/kernel/mm/hugepages/hugepages-1048576kB# cat >>>> nr_hugepages >>>> 0 >>>> root@ubuntu-15-10:/sys/kernel/mm/hugepages/hugepages-1048576kB# cat >>>> nr_hugepages_mempolicy >>>> 0 >>>> root@ubuntu-15-10:/sys/kernel/mm/hugepages/hugepages-1048576kB# cat >>>> nr_overcommit_hugepages >>>> 0 >>>> root@ubuntu-15-10:/sys/kernel/mm/hugepages/hugepages-1048576kB# cat >>>> resv_hugepages >>>> 0 >>>> root@ubuntu-15-10:/sys/kernel/mm/hugepages/hugepages-1048576kB# cat >>>> surplus_hugepages >>>> 0 >>>> root@ubuntu-15-10:/sys/kernel/mm/hugepages/hugepages-1048576kB# >>>> >>>> On Tue, May 10, 2016 at 6:19 PM, nousi <[email protected] <mailto: >>>> [email protected]>> wrote: >>>> >>>> yes, there are two directories under "/sys/kernel/mm/hugepages" >>>> >>>> root@ubuntu-15-10:/sys/kernel/mm/hugepages# ls -l >>>> total 0 >>>> drwxr-xr-x 2 root root 0 May 10 18:15 hugepages-1048576kB >>>> drwxr-xr-x 2 root root 0 May 10 18:15 hugepages-2048kB >>>> >>>> >>>> On Tue, May 10, 2016 at 6:13 PM, Maxim Uvarov >>>> <[email protected] <mailto:[email protected]>> wrote: >>>> >>>> In my case there is 1 Gb huge page in: >>>> /sys/kernel/mm/hugepages >>>> >>>> and odp_shm_reserve() just rounds all allocations to 1 GB: >>>> >>>> #ifdef MAP_HUGETLB >>>> huge_sz = odp_sys_huge_page_size(); <--- 1 GB here >>>> need_huge_page = (huge_sz && alloc_size > page_sz); >>>> /* munmap for huge pages requires sizes round up by page */ >>>> alloc_hp_size = (size + align + (huge_sz - 1)) & (-huge_sz); >>>> #endif >>>> >>>> Do you see the same thing? >>>> >>>> Maxim. >>>> >>>> On 05/10/16 15:07, nousi wrote: >>>> >>>> Hi Maxim, >>>> >>>> Thanks for your support. >>>> >>>> mamp return below error; >>>> "mmap: Cannot allocate memory" >>>> "mount -t hugetlbfs none /mnt/hugetlbfs" also does not help. >>>> >>>> Huge page allocation success in below two calls after that >>>> it fails. >>>> 1) odp_thread_globals >>>> 2) odp_buffer_pools >>>> >>>> Please have a look at the console logs below. >>>> >>>> odp_shared_memory.c:299:odp_shm_reserve(): >>>> odp_thread_globals: huge page allocation success ! >>>> odp_shared_memory.c:299:odp_shm_reserve(): >>>> odp_buffer_pools: huge page allocation success ! >>>> odp_pool.c:104:odp_pool_init_global(): >>>> Pool init global >>>> odp_pool.c:105:odp_pool_init_global(): pool_entry_s size >>>> 8512 >>>> odp_pool.c:106:odp_pool_init_global(): pool_entry_t size >>>> 8512 >>>> odp_pool.c:107:odp_pool_init_global(): odp_buffer_hdr_t >>>> size 216 >>>> odp_pool.c:108:odp_pool_init_global(): >>>> *mmap: Cannot allocate memory* >>>> odp_queue.c:130:odp_queue_init_global():Queue init ... >>>> odp_shared_memory.c:297:odp_shm_reserve(): odp_queues: >>>> No huge pages, fall back to normal pages, >>>> check: /proc/sys/vm/nr_hugepages. >>>> >>>> >>>> Recently I Moved from Ubuntu 14.04 to 15.10, in Ubuntu >>>> 14.04 I had run calssfier example without any huge page >>>> allocation error. >>>> >>>> On Tue, May 10, 2016 at 3:50 PM, Maxim Uvarov >>>> <[email protected] <mailto:[email protected]> >>>> <mailto:[email protected] >>>> <mailto:[email protected]>>> wrote: >>>> >>>> Does: >>>> mount -t hugetlbfs none /mnt/hugetlbfs >>>> >>>> help? >>>> >>>> Maxim. >>>> >>>> >>>> On 10 May 2016 at 13:16, Maxim Uvarov >>>> <[email protected] <mailto:[email protected]> >>>> <mailto:[email protected] >>>> <mailto:[email protected]>>> wrote: >>>> >>>> looks like you have enough free HP. Which error >>>> returns >>>> mmap() on try with HP? >>>> >>>> On 10 May 2016 at 11:57, nousi >>>> <[email protected] <mailto:[email protected]> >>>> <mailto:[email protected] >>>> >>>> <mailto:[email protected]>>> wrote: >>>> >>>> linaro@ubuntu-15-10:~$ cat >>>> /proc/sys/vm/nr_hugepages >>>> 1024 >>>> linaro@ubuntu-15-10:~$ >>>> >>>> linaro@ubuntu-15-10:~$ cat /proc/meminfo >>>> MemTotal: 8061836 kB >>>> MemFree: 470516 kB >>>> MemAvailable: 1901932 kB >>>> Buffers: 92600 kB >>>> Cached: 1939696 kB >>>> SwapCached: 7516 kB >>>> Active: 3238960 kB >>>> Inactive: 1804492 kB >>>> Active(anon): 2712440 kB >>>> Inactive(anon): 1069492 kB >>>> Active(file): 526520 kB >>>> Inactive(file): 735000 kB >>>> Unevictable: 76 kB >>>> Mlocked: 76 kB >>>> SwapTotal: 16547836 kB >>>> SwapFree: 16370160 kB >>>> Dirty: 19816 kB >>>> Writeback: 0 kB >>>> AnonPages: 3004784 kB >>>> Mapped: 679960 kB >>>> Shmem: 770776 kB >>>> Slab: 264692 kB >>>> SReclaimable: 212172 kB >>>> SUnreclaim: 52520 kB >>>> KernelStack: 11952 kB >>>> PageTables: 65780 kB >>>> NFS_Unstable: 0 kB >>>> Bounce: 0 kB >>>> WritebackTmp: 0 kB >>>> CommitLimit: 19530176 kB >>>> Committed_AS: 11165432 kB >>>> VmallocTotal: 34359738367 kB >>>> VmallocUsed: 410416 kB >>>> VmallocChunk: 34358947836 kB >>>> HardwareCorrupted: 0 kB >>>> AnonHugePages: 583680 kB >>>> CmaTotal: 0 kB >>>> CmaFree: 0 kB >>>> HugePages_Total: 1024 >>>> HugePages_Free: 1022 >>>> HugePages_Rsvd: 1022 >>>> HugePages_Surp: 0 >>>> Hugepagesize: 2048 kB >>>> DirectMap4k: 234400 kB >>>> DirectMap2M: 6991872 kB >>>> DirectMap1G: 2097152 kB >>>> linaro@ubuntu-15-10:~$ >>>> >>>> >>>> On Tue, May 10, 2016 at 12:38 PM, Maxim Uvarov >>>> <[email protected] >>>> <mailto:[email protected]> >>>> <mailto:[email protected] >>>> <mailto:[email protected]>>> >>>> wrote: >>>> >>>> Please put output for: >>>> cat /proc/meminfo >>>> cat /proc/sys/vm/nr_hugepages >>>> >>>> Thank you, >>>> Maxim. >>>> >>>> >>>> >>>> On 10 May 2016 at 08:36, nousi >>>> <[email protected] <mailto:[email protected]> >>>> <mailto:[email protected] >>>> <mailto:[email protected]>>> wrote: >>>> >>>> mmap is failing in "odp_shm_reserve" >>>> function >>>> (odp_queue_init_global() ---> >>>> odp_shm_reserve() >>>> ---> odp_shm_reserve()) >>>> >>>> >>>> debug logs: >>>> >>>> root@ubuntu-15-10 >>>> :/home/linaro/linaro/odp/example/classifier# >>>> ./odp_classifier -i eno1 -m 0 -p >>>> "ODP_PMR_SIP_ADDR:192.168.10.11 >>>> <tel:192.168.10.11>:FFFFFFFF:queue1" >>>> -p "ODP_PMR_SIP_ADDR:10.130.69.0 >>>> <http://10.130.69.0>:000000FF:queue2" >>>> -p "ODP_PMR_SIP_ADDR:10.130.68.0 >>>> <http://10.130.68.0>:FFFFFE00:queue3" >>>> >>>> odp_pool.c:104:odp_pool_init_global(): >>>> Pool init global >>>> odp_pool.c:105:odp_pool_init_global(): >>>> pool_entry_s size 8512 >>>> odp_pool.c:106:odp_pool_init_global(): >>>> pool_entry_t size 8512 >>>> odp_pool.c:107:odp_pool_init_global(): >>>> odp_buffer_hdr_t size 216 >>>> odp_pool.c:108:odp_pool_init_global(): >>>> odp_queue.c:130:odp_queue_init_global():Queue init >>>> ... >>>> odp_shared_memory.c:296:odp_shm_reserve():odp_queues: >>>> :: No huge pages, fall back to >>>> normal pages, >>>> check: /proc/sys/vm/nr_hugepages. >>>> odp_queue.c:154:odp_queue_init_global():done >>>> odp_queue.c:155:odp_queue_init_global():Queue init >>>> global >>>> odp_queue.c:157:odp_queue_init_global(): struct >>>> queue_entry_s size 320 >>>> odp_queue.c:159:odp_queue_init_global(): >>>> queue_entry_t size 320 >>>> odp_queue.c:160:odp_queue_init_global(): >>>> odp_schedule.c:145:odp_schedule_init_global():Schedule >>>> init ... >>>> odp_shared_memory.c:296:odp_shm_reserve():odp_scheduler: >>>> :: No huge pages, fall back to >>>> normal pages, >>>> check: /proc/sys/vm/nr_hugepages. >>>> odp_shared_memory.c:296:odp_shm_reserve():odp_sched_pool: >>>> :: No huge pages, fall back to >>>> normal pages, >>>> check: /proc/sys/vm/nr_hugepages. >>>> odp_schedule.c:226:odp_schedule_init_global():done >>>> odp_shared_memory.c:296:odp_shm_reserve():odp_pktio_entries: >>>> :: No huge pages, fall back to >>>> normal pages, >>>> check: /proc/sys/vm/nr_hugepages. >>>> odp_shared_memory.c:296:odp_shm_reserve():crypto_pool: >>>> :: No huge pages, fall back to >>>> normal pages, >>>> check: /proc/sys/vm/nr_hugepages. >>>> odp_shared_memory.c:296:odp_shm_reserve():shm_odp_cos_tbl: >>>> :: No huge pages, fall back to >>>> normal pages, >>>> check: /proc/sys/vm/nr_hugepages. >>>> odp_shared_memory.c:296:odp_shm_reserve():shm_odp_pmr_tbl: >>>> :: No huge pages, fall back to >>>> normal pages, >>>> check: /proc/sys/vm/nr_hugepages. >>>> main :: odp_init_global done! >>>> odp_classifier.c:500:main():main :: odp_init_local >>>> done! >>>> odp_classifier.c:505:main():main :: >>>> odp_shm_reserve done! >>>> >>>> ODP system info >>>> --------------- >>>> ODP API version: 1.10.0 >>>> CPU model: Intel(R) Core(TM) i7-5600U >>>> CPU >>>> CPU freq (hz): 2600000000 >>>> Cache line size: 64 >>>> CPU count: 4 >>>> >>>> Running ODP appl: "odp_classifier" >>>> ----------------- >>>> Using IF:eno1 >>>> >>>> num worker threads: 2 >>>> first CPU: 2 >>>> cpu mask: 0xC >>>> odp_shared_memory.c:296:odp_shm_reserve():packet_pool: >>>> :: No huge pages, fall back to >>>> normal pages, >>>> check: /proc/sys/vm/nr_hugepages. >>>> odp_pool.c:759:odp_pool_print():Pool info >>>> odp_pool.c:760:odp_pool_print():--------- >>>> odp_pool.c:762:odp_pool_print(): pool 1 >>>> odp_pool.c:764:odp_pool_print(): name packet_pool >>>> odp_pool.c:769:odp_pool_print(): pool type packet >>>> odp_pool.c:771:odp_pool_print(): pool storage ODP >>>> managed shm handle 11 >>>> odp_pool.c:773:odp_pool_print(): pool status active >>>> odp_pool.c:777:odp_pool_print(): pool opts >>>> segmented, non-zeroized, created >>>> odp_pool.c:778:odp_pool_print(): pool base >>>> 0x7f5091aab000 >>>> odp_pool.c:780:odp_pool_print(): pool size 1310720 >>>> (320 pages) >>>> odp_pool.c:781:odp_pool_print(): pool mdata base >>>> 0x7f5091bb5940 >>>> odp_pool.c:782:odp_pool_print(): udata size 0 >>>> odp_pool.c:783:odp_pool_print(): headroom 66 >>>> odp_pool.c:784:odp_pool_print(): tailroom 0 >>>> odp_pool.c:791:odp_pool_print(): seg length 1856 >>>> requested, 1936 used >>>> odp_pool.c:793:odp_pool_print(): pkt length 1856 >>>> requested, 1936 used >>>> odp_pool.c:795:odp_pool_print(): num bufs 564 >>>> odp_pool.c:797:odp_pool_print(): bufs available 564 >>>> odp_pool.c:798:odp_pool_print(): bufs in use 0 >>>> odp_pool.c:799:odp_pool_print(): buf allocs 0 >>>> odp_pool.c:800:odp_pool_print(): buf frees 0 >>>> odp_pool.c:801:odp_pool_print(): buf empty 0 >>>> odp_pool.c:803:odp_pool_print(): blk size 1936 >>>> odp_pool.c:805:odp_pool_print(): blks available 564 >>>> odp_pool.c:806:odp_pool_print(): blk allocs 0 >>>> odp_pool.c:807:odp_pool_print(): blk frees 0 >>>> odp_pool.c:808:odp_pool_print(): blk empty 0 >>>> odp_pool.c:809:odp_pool_print(): buf high wm >>>> value 282 >>>> odp_pool.c:810:odp_pool_print(): buf high wm count 0 >>>> odp_pool.c:811:odp_pool_print(): buf low wm >>>> value 141 >>>> odp_pool.c:812:odp_pool_print(): buf low wm count 0 >>>> odp_pool.c:813:odp_pool_print(): blk high wm >>>> value 282 >>>> odp_pool.c:814:odp_pool_print(): blk high wm count 0 >>>> odp_pool.c:815:odp_pool_print(): blk low wm >>>> value 141 >>>> odp_pool.c:816:odp_pool_print(): blk low wm count 0 >>>> main :: odp_pool_print done! >>>> odp_packet_io.c:230:setup_pktio_entry():eno1 uses >>>> socket_mmap >>>> created pktio:01, dev:eno1, queue >>>> mode (ATOMIC >>>> queues) >>>> default pktio01 >>>> odp_shared_memory.c:296:odp_shm_reserve(): >>>> DefaultPool: >>>> :: No huge pages, fall back to >>>> normal pages, >>>> check: /proc/sys/vm/nr_hugepages. >>>> odp_shared_memory.c:296:odp_shm_reserve(): >>>> queue1Pool0: >>>> :: No huge pages, fall back to >>>> normal pages, >>>> check: /proc/sys/vm/nr_hugepages. >>>> odp_shared_memory.c:296:odp_shm_reserve(): >>>> queue2Pool1: >>>> :: No huge pages, fall back to >>>> normal pages, >>>> check: /proc/sys/vm/nr_hugepages. >>>> odp_shared_memory.c:296:odp_shm_reserve(): >>>> queue3Pool2: >>>> :: No huge pages, fall back to >>>> normal pages, >>>> check: /proc/sys/vm/nr_hugepages. >>>> >>>> ---------------------------------------- >>>> CLASSIFIER EXAMPLE STATISTICS >>>> ---------------------------------------- >>>> CONFIGURATION >>>> >>>> COS VALUE MASK >>>> ---------------------------------------- >>>> queue1 192.168.10.11 >>>> <tel:192.168.10.11> FFFFFFFF >>>> queue2 10.130.69.0 000000FF >>>> queue3 10.130.68.0 FFFFFE00 >>>> >>>> RECEIVED PACKETS >>>> ---------------------------------------- >>>> queue1 |queue2 |queue3 |DefaultCos >>>> |Total Packets >>>> queue pool |queue pool |queue pool >>>> |queue pool | >>>> 845 845 |0 0 |0 0 |38 >>>> 38 |883 >>>> >>>> >>>> On Mon, May 9, 2016 at 9:00 PM, Bill >>>> Fischofer >>>> <[email protected] >>>> <mailto:[email protected]> >>>> <mailto:[email protected] >>>> <mailto:[email protected]>>> wrote: >>>> >>>> >>>> >>>> On Mon, May 9, 2016 at 6:57 AM, >>>> nousi >>>> <[email protected] >>>> <mailto:[email protected]> >>>> <mailto:[email protected] >>>> <mailto:[email protected]>>> wrote: >>>> >>>> >>>> Hi All, >>>> >>>> Pleas help me in running ODP >>>> classifier >>>> example with huge pages. >>>> In ubuntu 15.10 by default >>>> interface >>>> naming as "eno1" and value in >>>> "/proc/sys/vm/nr_hugepages." is 1024. >>>> classifier example program >>>> could not able >>>> to use huge pages even though >>>> nr_hugepages >>>> value is non zero. >>>> I could able to run the >>>> classier example, >>>> but it is not using huge pages. >>>> /* >>>> console log is pasted below >>>> for you reference. >>>> >>>> */root@odp/example/classifier# >>>> >>>> ./odp_classifier -i eno1 -m 0 -p >>>> "ODP_PMR_SIP_ADDR:192.168.10.11 >>>> <tel:192.168.10.11>:FFFFFFFF:queue1" >>>> -p >>>> "ODP_PMR_SIP_ADDR:10.130.69.0 >>>> <http://10.130.69.0>:000000FF:queue2" >>>> -p >>>> "ODP_PMR_SIP_ADDR:10.130.68.0 >>>> <http://10.130.68.0>:FFFFFE00:queue3" >>>> >>>> odp_pool.c:104:odp_pool_init_global(): >>>> Pool init global >>>> odp_pool.c:105:odp_pool_init_global(): >>>> pool_entry_s size 8512 >>>> odp_pool.c:106:odp_pool_init_global(): >>>> pool_entry_t size 8512 >>>> odp_pool.c:107:odp_pool_init_global(): >>>> odp_buffer_hdr_t size 216 >>>> odp_pool.c:108:odp_pool_init_global(): >>>> odp_queue.c:130:odp_queue_init_global():Queue >>>> init ... >>>> odp_shared_memory.c:296:odp_shm_reserve(): >>>> odp_queues: >>>> No huge pages, fall back >>>> to normal pages, >>>> check: >>>> /proc/sys/vm/nr_hugepages. >>>> >>>> >>>> This is an informational message >>>> saying that >>>> the linux-generic implementation >>>> was unable to >>>> allocate huge pages so it's >>>> falling back to >>>> normal pages. I'm not sure why >>>> you're seeing >>>> that except that it seems that some >>>> allocations may have been >>>> successful (those in >>>> odp_pool.c) while those for queue >>>> initialization were not. >>>> >>>> I'll let others who are more >>>> expert in this >>>> area chime in with some additional >>>> thoughts. >>>> >>>> >>>> >>>> >>>> Thanks & Regards, >>>> /B.Nousilal, >>>> / >>>> >>>> _______________________________________________ >>>> lng-odp mailing list >>>> [email protected] <mailto:[email protected]> >>>> <mailto: >>>> [email protected] >>>> <mailto:[email protected]>> >>>> https://lists.linaro.org/mailman/listinfo/lng-odp >>>> >>>> >>>> >>>> >>>> >>>> -- /Thanks & Regards, >>>> B.Nousilal, >>>> / >>>> >>>> _______________________________________________ >>>> lng-odp mailing list >>>> [email protected] <mailto:[email protected]> >>>> <mailto:[email protected] >>>> <mailto:[email protected]>> >>>> https://lists.linaro.org/mailman/listinfo/lng-odp >>>> >>>> >>>> >>>> >>>> >>>> -- /Thanks & Regards, >>>> B.Nousilal, >>>> / >>>> >>>> >>>> >>>> >>>> >>>> >>>> -- /Thanks & Regards, >>>> B.Nousilal, >>>> / >>>> >>>> >>>> >>>> >>>> >>>> -- /Thanks & Regards, >>>> B.Nousilal, >>>> / >>>> >>>> >>>> >>>> >>>> -- >>>> /Thanks & Regards, >>>> B.Nousilal, >>>> / >>>> >>> >>> >> >> >> -- >> >> >> *Thanks & Regards,B.Nousilal,* >> > > -- *Thanks & Regards,B.Nousilal,*
_______________________________________________ lng-odp mailing list [email protected] https://lists.linaro.org/mailman/listinfo/lng-odp
