[smartos-discuss] zpool import hangs

2014-04-03 Thread Youzhong Yang
We have 3 test servers running SmartOS, host B and C have zpools zp01 and zp02 respectively, host A can see disks of zp01 and zp02 (disks of zp01 are connected to HBA ports of A and B; disks of zp02 are connected to HBA ports of A and C). And I have a loop to test zpool import/export: for i in

[smartos-discuss] Re: [developer] zpool import hangs

2014-04-04 Thread Youzhong Yang
Good, here is the bug report I just filed: https://www.illumos.org/issues/4730 On Fri, Apr 4, 2014 at 5:17 PM, Irek Szczesniak iszczesn...@gmail.comwrote: On Fri, Apr 4, 2014 at 11:08 PM, Youzhong Yang youzh...@gmail.com wrote: Thanks Robert, surya and everyone, you guys rock! I built

[smartos-discuss] How to use mdb to view process stack?

2014-04-15 Thread Youzhong Yang
I am interested in what a running process is doing, i.e. its user stack. I know pstack can be used to print process stack, but sometimes it may not work as you expected. Is it possible to use mdb to view process stack? There's a blog article 'use mdb to view process stack' created by some

[smartos-discuss] Re: [developer] NVMe SSDs (Intel DC P3700, P3600) won't show up

2016-06-22 Thread Youzhong Yang
driver somehow? > > If it's reproducible as part of attaching, it might be interesting to > see if we can figure out what accesses to these devices is causing that, > which may be an issue we're not properly handling with the device or a > driver bug somewhere. > > Robert > >

[smartos-discuss] Re: Is it possible to roll back a ZPOOL(which cannot be imported) to its last known good state?

2016-01-26 Thread Youzhong Yang
abel); return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, ENXIO)); } Fortunately we had backup of the zpool available so we just restored from the backup. Thanks again for the tips which may be useful in the future (I hope we will not make such mistake again). -Youzhong On Mon, Jan 25, 2016 at 3:50

Re: [smartos-discuss] [developer] NVMe SSDs (Intel DC P3700, P3600) won't show up

2016-08-01 Thread Youzhong Yang
"uint32_t csts_rsvd:28;" be "uint32_t csts_rsvd:27;"? Thanks! On Mon, Aug 1, 2016 at 2:59 PM, Youzhong Yang <youzh...@gmail.com> wrote: > I'm not sure what sector size it uses. Randomly failing some devices > suggests that the NVMe driver is not doing the right

Re: [smartos-discuss] [developer] NVMe SSDs (Intel DC P3700, P3600) won't show up

2016-08-01 Thread Youzhong Yang
n, Aug 01, 2016 at 03:32:50PM -0400, Youzhong Yang wrote: > > This does not make sense to me, not sure if it's relevant to the issue I > am > > seeing: > > > > http://www.nvmexpress.org/wp-content/uploads/NVM-Express-1_1b.pdf > > page 42 defines 'CSTS -Contro

Re: [smartos-discuss] [developer] NVMe SSDs (Intel DC P3700, P3600) won't show up

2016-08-11 Thread Youzhong Yang
nt.com> wrote: > On 8/4/16 11:25 , Youzhong Yang wrote: > > Thanks for the input Robert. > > > > I believe the issue is now resolved by using MSI-X (instead of FIXED) > > interrupt type inside nvme_init() for the admin queue. > > > > Here is the issue repo

[smartos-discuss] TSX-NI support in illumos?

2016-07-21 Thread Youzhong Yang
Just curious, is TSX supported in illumos? "*Transactional Synchronization Extensions* (*TSX*) is an extension to the x86 instruction set architecture (ISA) that adds hardware transactional memory

Re: [developer] Re: [smartos-discuss] TSX-NI support in illumos?

2016-07-21 Thread Youzhong Yang
> something I'd turn to. > > Robert > > >> On Jul 21, 2016, at 7:34 AM, Youzhong Yang <youzh...@gmail.com> wrote: > >> > >> Just curious, is TSX supported in illumos? > >> > >> "Transactional Synchronization Extensions (TSX) is an ext

Re: [smartos-discuss] clock_gettime() causes sporadic segmentation fault

2016-07-20 Thread Youzhong Yang
I verified that the issue is fixed now. Thank you! On Tue, Jul 19, 2016 at 5:19 PM, Patrick Mooney <patrick.moo...@joyent.com> wrote: > With the fix for OS-5515 merged, this should be solved. > > Thanks for bringing it to our attention. > > On 18 July 2016 at 08:50,

Re: [smartos-discuss] [developer] NVMe SSDs (Intel DC P3700, P3600) won't show up

2016-08-01 Thread Youzhong Yang
nks, -Youzhong On Fri, Jun 24, 2016 at 8:13 PM, Robert Mustacchi <r...@joyent.com> wrote: > On 6/24/16 11:05 , Youzhong Yang wrote: > > > I panicked the host when e_ddi_retire_device() is called, here is what I > > found: > > > > it is /usr/lib/fm/fm

[smartos-discuss] vioblk driver

2017-01-30 Thread Youzhong Yang
Hi All, A quick question - how can I make use of vioblk device driver? Creating a SmartOS VM using Linux KVM? The reason I am asking this is that I want to test a change in blkdev, to exercise some code path in it which can't be done using NVMe devices, such as the 'dump' functionality. Thanks,

Re: [smartos-discuss] vioblk driver

2017-01-30 Thread Youzhong Yang
1454 int sc_blk_size = 0x200 int sc_pblk_size = 0x200 Quite confused .. Any thoughts? Thanks, --Youzhong On Mon, Jan 30, 2017 at 10:10 AM, Nahum Shalman <nshal...@omniti.com> wrote: > You can also create a SmartOS KVM VM on a SmartOS machine with a virtio > virtual disk. >

[smartos-discuss] cannot open '...': dataset does not exist

2017-02-13 Thread Youzhong Yang
Really weird. I am seeing this issue with the latest image, but not with the OS image built in last November. On a freshly installed VM, there's no problem. # zfs create clusters/vmgr/test cannot open 'vmgr/test': dataset does not exist # echo $? 0 # zfs list clusters/vmgr/test NAME

Re: [smartos-discuss] smartos ssd disk question

2017-01-23 Thread Youzhong Yang
it is power safe and we've tested it here. https://toshiba.semicon-storage.com/us/product/storage-products/enterprise-ssd/px02smb-px02smfxxx.html?sug=1 On Mon, Jan 23, 2017 at 12:01 PM, Robert Mustacchi <r...@joyent.com> wrote: > On 1/23/17 6:29 , Youzhong Yang wrote: > > Add

Re: [smartos-discuss] smartos ssd disk question

2017-01-23 Thread Youzhong Yang
I don't know. What we know is that the sync write performance of zil drive is awful if cache-nonvolatile:true is not set. On Mon, Jan 23, 2017 at 12:30 PM, Robert Mustacchi <r...@joyent.com> wrote: > On 1/23/17 9:20 , Youzhong Yang wrote: > > it is power safe and we've

Re: [smartos-discuss] how to block init 6 on global zone

2016-09-29 Thread Youzhong Yang
In smartos-live/overlay/generic/lib/svc/method/fs-joyent, if you add + if [[ -d /usbkey/crud ]]; then + (cd /usbkey/crud;find . -type f -print|xargs -i sh -c 'cd /;mkdir -p `dirname "{}"`;touch "{}";mount -F lofs /usbkey/crud/"{}" "{}"') + fi then all files under /usbkey/crud will be lofs mouted

[smartos-discuss] zrlock issue: in LOCKED state and never released

2016-09-19 Thread Youzhong Yang
Hi, One of our servers had hundreds of threads waiting for a zrlock to be released, but it never did: http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/fs/zfs/zrlock.c#96 I am just wondering how this could happen, the only place trying to lock it seems to be this one:

[smartos-discuss] Re: build failure related to python version

2016-09-30 Thread Youzhong Yang
/python2.6 On Fri, Sep 30, 2016 at 4:01 PM, Youzhong Yang <youzh...@gmail.com> wrote: > I am trying to build SmartOS today, pulling illumos-joyent from github.com, > here is what I encountered: > > nightly.log: > sh: line 1: /root/smartos-live/projects/illumos/usr/src/tools/

[smartos-discuss] build failure related to python version

2016-09-30 Thread Youzhong Yang
I am trying to build SmartOS today, pulling illumos-joyent from github.com, here is what I encountered: nightly.log: sh: line 1: /root/smartos-live/projects/illumos/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/hdrchk: not found *** Error code 127 # head -1

Re: [smartos-discuss] Re: build failure related to python version

2016-09-30 Thread Youzhong Yang
Now it works. Thanks! On Fri, Sep 30, 2016 at 4:39 PM, Robert Mustacchi <r...@joyent.com> wrote: > On 9/30/16 13:05 , Youzhong Yang wrote: > > Sorry, the 'head' output is after I manually modified hdrchk script. Its > > first line was referring to /usr/bin/python2.6. > &

[smartos-discuss] build: 00000000T000000Z

2017-04-06 Thread Youzhong Yang
Hi, I noticed something strange today: # ssh host uname -a SunOS batfs9920 5.11 joyent_20170406T161321Z i86pc i386 i86pc # ssh host - SmartOS (build: T00Z) By the way, I built the image today after merging latest stuff from illumos-joyent, smartos-live etc. Thanks, --Youzhong

Re: [smartos-discuss] To the OmniOS, OI and SmartOS community

2017-05-15 Thread Youzhong Yang
I would recommend Plan C. We've used SmartOS for years, with hundreds of servers as storage backend, serving thousands of Linux/Mac/Windows clients. Joyent has the best talented engineers and they are always willing to help. By the way, we don't care much about packages, a stable and reliable

[smartos-discuss] rpcbind memory leaks

2017-05-22 Thread Youzhong Yang
Hi all, I am just wondering if you have seen this same issue. On one of our servers, the rpcbind daemon used 3.6G memory; on another server, it reported out of memory error: svc_vc_create: makefd_xprt : out of memory do_accept : out of memory xdrrec_create: out of memory svc_fd_xprtcopy:

Re: [smartos-discuss] service varpd not working

2018-05-18 Thread Youzhong Yang
; > On May 18, 2018, at 5:11 PM, Youzhong Yang <youzh...@gmail.com> wrote: > > > > Thanks Jason. In my environment, I've disabled KPTI through boot var > 'kpti=0', not sure if it contributes to the issue. > > kpti shouldn't affect anything. > > Could you:

Re: [smartos-discuss] service varpd not working

2018-05-18 Thread Youzhong Yang
gt; recreate this locally. > > > From: Youzhong Yang <youzh...@gmail.com> <youzh...@gmail.com> > Reply: smartos-discuss@lists.smartos.org <smartos-discuss@lists. > smartos.org> <smartos-discuss@lists.smartos.org> > Date: May 18, 2018 at 3:04:31 PM &g

Re: [smartos-discuss] service varpd not working

2018-05-18 Thread Youzhong Yang
setup. I’ll update the list once I’ve figured out what’s going on. > > > From: Youzhong Yang <youzh...@gmail.com> <youzh...@gmail.com> > Reply: smartos-discuss@lists.smartos.org <smartos-discuss@lists. > smartos.org> <smartos-discuss@lists.smartos.org> >

[smartos-discuss] service varpd not working

2018-05-18 Thread Youzhong Yang
Hi, Today I built SmartOS from the master branch. After rebooting the server, varpd did not start up: # svcs varpd STATE STIMEFMRI maintenance15:53:23 svc:/network/varpd:default # tail /var/svc/log/network-varpd\:default.log [ May 18 15:53:23 Executing start method

[smartos-discuss] diet too much = suicide ?

2018-03-17 Thread Youzhong Yang
Last few days I started evaluating the most recent build which has KPTI/PCID fix. Thanks to the excellent work done by Joyent. So far I've seen no issue, except the expected performance degradation. But there's one thing that caught my eye. A dozen of our perl scripts which depend on /usr/perl5

[smartos-discuss] fall out due to OS-5726 (bump pid max to a reasonable modern value)

2018-03-18 Thread Youzhong Yang
We set pidmax in /etc/system to 9, and our application allocates shared memory based on the pidmax. Due to OS-5726 ( bump pid max to a reasonable modern value): https://smartos.org/bugview/OS-5726 https://github.com/joyent/illumos-joyent/commit/43b3bab58788e30af95d6dd211800182624ede96 It

Re: [smartos-discuss] still hang at boot - OS-7079 mp_startup_common races itself

2018-08-13 Thread Youzhong Yang
> bug in OS-7079, so I’ll see if I can get it to hang (though it’ll take a > bit to do a full build). > > > From: Youzhong Yang > Reply: Youzhong Yang > Date: August 13, 2018 at 12:27:52 AM > > To: Jason King > Cc: smartos-discuss@lists.smartos.org > > Subject:

[smartos-discuss] still hang at boot - OS-7079 mp_startup_common races itself

2018-08-12 Thread Youzhong Yang
Today I built a smartos image (with all git repos synced to master) and rebooted the host with that image. It hung after the banner message + one more line about power management or something. Then I reverted OS-7079, built an image, rebooted, it worked perfectly. So does it mean OS-7079 fixed

Re: [smartos-discuss] still hang at boot - OS-7079 mp_startup_common races itself

2018-08-12 Thread Youzhong Yang
sy to > trigger until recent CPUs. > > > From: Youzhong Yang > Reply: smartos-discuss@lists.smartos.org smartos.org> > Date: August 12, 2018 at 10:46:05 PM > To: smartos-discuss@lists.smartos.org > > Subject: [smartos-discuss] still hang at boot - OS-7079 >

Re: [smartos-discuss] still hang at boot - OS-7079 mp_startup_common races itself

2018-08-12 Thread Youzhong Yang
t; > It’s certainly possible it could have revealed another bug — OS-7079 > itself was introduced almost 10 years ago, but didn’t seem to be easy to > trigger until recent CPUs. > > > From: Youzhong Yang > Reply: smartos-discuss@lists.smartos.org smartos.org>

Re: [smartos-discuss] still hang at boot - OS-7079 mp_startup_common races itself

2018-08-12 Thread Youzhong Yang
lic’ in > there. > > Try > > https://us-east.manta.joyent.com/jbk/public/OS-7079/ > platform-20180719T001516Z.iso > > > From: Youzhong Yang > Reply: Youzhong Yang > Date: August 13, 2018 at 12:12:52 AM > > To: Jason King > Cc: smartos-discuss@lists.smartos.org &g

[smartos-discuss] zvol image on image server

2018-06-27 Thread Youzhong Yang
If I need to store our own kvm image on our server, what format of the image file should be? zfs send stream and compressed by gzip? where to put the file? together with the manifest file? The following image on joyent server does not say anything about the file location. I have no idea what it

Re: [smartos-discuss] zvol image on image server

2018-06-28 Thread Youzhong Yang
ux-prepare-image at > https://raw.githubusercontent.com/joyent/sdc-imgapi/master/ > tools/prepare-image/linux-prepare-image > > You can then install that image on the server with > > imgadm install -f $name-$version.zvol -m $name-$version.imgmanifest > > HTH, > Mike > > > On Wed

Re: [smartos-discuss] still hang at boot - OS-7079 mp_startup_common races itself

2018-08-14 Thread Youzhong Yang
with just the above change I can rebase/checkout/etc > as needed) — though maybe those with better mdb-fu than me have better luck > with that approach. > > > From: Youzhong Yang > Reply: Youzhong Yang > Date: August 13, 2018 at 9:29:07 AM > > To: Jason King > Cc: sma

Re: [smartos-discuss] ACPI Testing Request

2018-08-16 Thread Youzhong Yang
Supermicro X11DPU, BIOS 2.1, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz No issue. On Thu, Aug 16, 2018 at 12:01 PM, Robert Mustacchi wrote: > Hi All, > > There have been a number of issues with boot hangs on some of the more > recent Kaby Lake processors and a few Skylake SKUs. We were able to

[smartos-discuss] iostat hangs

2014-06-02 Thread Youzhong Yang via smartos-discuss
Hi All, On one of our servers running SmartOS, the 'iostat' hung and there're a bunch of stuck 'iostat' commands. It seems everyone of them has the same kernel stack: stack pointer for thread ffbe78d0f3a0: ff0173d84380 [ ff0173d84380 _resume_from_idle+0xf4() ] ff0173d843b0

[smartos-discuss] Re: [developer] Issue 4965: NLM: Mac OS X client can be stuck at locking a NFS file on an illumos server (SmartOS, Nexenta or OmniOS etc.) running open source lockd

2014-07-11 Thread Youzhong Yang via smartos-discuss
||\ (_status) == RPC_XPRTFAILED) static struct kmem_cache *nlm_rpch_cache = NULL; On Tue, Jul 8, 2014 at 2:44 PM, Marcel Telka mar...@telka.sk wrote: Hi, Nice work. Please find my comments inline. On Tue, Jul 08, 2014 at 01:43:36PM -0400, Youzhong Yang via illumos-developer wrote

[smartos-discuss] Issue with LSI 3108 MegaRAID ROMB card

2014-07-17 Thread Youzhong Yang via smartos-discuss
Hi All, We have problem using the LSI 3108 card, just wondering if anyone here has any success story using this card in production. Here is the FM version info and error we got in /var/adm/messages: BIOS Version : 6.13.00_4.14.05.00_0x06010600 Ctrl-R Version : 5.01-0004 FW Version

[smartos-discuss] Re: [OmniOS-discuss] Issue with LSI 3108 MegaRAID ROMB card

2014-07-17 Thread Youzhong Yang via smartos-discuss
: On Jul 17, 2014, at 11:07 AM, Youzhong Yang youzh...@gmail.com wrote: Hi All, We have problem using the LSI 3108 card, just wondering if anyone here has any success story using this card in production. SNIP! When mr_sas(7d) was updated for 2208, it included untested 3108 support. 3108

Re: [smartos-discuss] Re: [OmniOS-discuss] Issue with LSI 3108 MegaRAID ROMB card

2014-07-17 Thread Youzhong Yang via smartos-discuss
Wesolowski keith.wesolow...@joyent.com wrote: On Thu, Jul 17, 2014 at 11:59:16AM -0400, Youzhong Yang via smartos-discuss wrote: We ordered these Supermicro X9DRW-CF/CTF boxes which have ROMB LSI 3108 on the motherboard and got stuck. We will probably add 9211-8i HBA cards to the machines

[smartos-discuss] epoll_ctl and epoll_wait on SmartOS

2014-07-30 Thread Youzhong Yang via smartos-discuss
Hi All, The man page of epoll_ctl says: The data field specifies the datum to be associated with the event and will be returned via epoll_wait(3C). The events field denotes both the desired events (when specified via epoll_ctl()) and the events that have

Re: [smartos-discuss] epoll_ctl and epoll_wait on SmartOS

2014-07-31 Thread Youzhong Yang via smartos-discuss
, Youzhong Yang via smartos-discuss smartos-discuss@lists.smartos.org wrote: Hi All, The man page of epoll_ctl says: The data field specifies the datum to be associated with the event and will be returned via epoll_wait(3C). The events field denotes both the desired

Re: [smartos-discuss] epoll_ctl and epoll_wait on SmartOS

2014-08-01 Thread Youzhong Yang via smartos-discuss
implementation. I have pushed the fix: https://github.com/joyent/illumos-joyent/commit/471f1f3ef02e25a84d7c36c857ed260ae2b7ac99 Thanks for taking the time for the clean test case -- and apologies for the bug! - Bryan On Thu, Jul 31, 2014 at 7:28 AM, Youzhong Yang youzh...@gmail.com

Re: [smartos-discuss] epoll_ctl and epoll_wait on SmartOS

2014-08-01 Thread Youzhong Yang via smartos-discuss
I got my new smartos image and alos samba 4.1.10 built, it works, no more 'net ads join' crashes. Your help is absolutely appreciated. Thanks, -Youzhong On Fri, Aug 1, 2014 at 9:42 AM, Youzhong Yang youzh...@gmail.com wrote: Thanks very much for the quick fix. I'll build a new image

[smartos-discuss] How to release reserved TCP port binding in BOUND state

2014-11-03 Thread Youzhong Yang via smartos-discuss
Hello, We are having a very strange issue on one of servers. The issue is that fcntl locking over NFS returns 'no locks available' immediately. dtrace shows that bindresvport() returns error code 125 (EADDRINUSE): # dtrace -n 'fbt:rpcmod:bindresvport:return /arg1 != 0/ {stack(); printf(ret =

[smartos-discuss] Re: [developer] How to release reserved TCP port binding in BOUND state

2014-11-04 Thread Youzhong Yang via smartos-discuss
/source/xref/illumos-gate/usr/src/uts/common/rpc/clnt_cots.c#508 Will dynamically changing this variable have any side effect? Thanks, -Youzhong On Mon, Nov 3, 2014 at 4:50 PM, Marcel Telka mar...@telka.sk wrote: Hi Youzhong, On Mon, Nov 03, 2014 at 04:14:55PM -0500, Youzhong Yang via illumos

Re: [smartos-discuss] inotify driver not loaded automatically

2014-12-09 Thread Youzhong Yang via smartos-discuss
Yeah, that's what I need. Thanks Robert. On Tue, Dec 9, 2014 at 11:04 AM, Robert Mustacchi r...@joyent.com wrote: On 12/9/14 8:02 , Youzhong Yang via smartos-discuss wrote: Hi All, I cherry-picked this cool inotify implementation ( https://github.com/joyent/illumos-joyent/commit

Re: [smartos-discuss] inotify driver not loaded automatically

2014-12-09 Thread Youzhong Yang via smartos-discuss
commits/changes do not introduce any issues. Thanks Keith. On Tue, Dec 9, 2014 at 11:06 AM, Keith Wesolowski keith.wesolow...@joyent.com wrote: On Tue, Dec 09, 2014 at 11:02:03AM -0500, Youzhong Yang via smartos-discuss wrote: I cherry-picked this cool inotify implementation ( https

[smartos-discuss] inotify issues

2014-12-10 Thread Youzhong Yang via smartos-discuss
Hello again, I downloaded smartos latest image (joyent_20141127T173954Z) and tested inotify using it. Two issues were observed: 1. ioctl(fd, FIONREAD, ...) does not work as expected. It does not return the number of bytes that are available to be read on an inotify instance. 2. In an OS zone,

Re: [smartos-discuss] inotify issues

2014-12-10 Thread Youzhong Yang via smartos-discuss
in in the specified argument. I'll get that fixed today... - Bryan On Wed, Dec 10, 2014 at 8:00 AM, Youzhong Yang via smartos-discuss smartos-discuss@lists.smartos.org wrote: Hello again, I downloaded smartos latest image (joyent_20141127T173954Z) and tested inotify using it. Two

Re: [smartos-discuss] inotify issues

2014-12-10 Thread Youzhong Yang via smartos-discuss
, Dec 10, 2014 at 8:00 AM, Youzhong Yang via smartos-discuss smartos-discuss@lists.smartos.org wrote: Hello again, I downloaded smartos latest image (joyent_20141127T173954Z) and tested inotify using it. Two issues were observed: 1. ioctl(fd, FIONREAD, ...) does not work as expected. It does

Re: [smartos-discuss] inotify issues

2014-12-11 Thread Youzhong Yang via smartos-discuss
On Wed, Dec 10, 2014 at 4:07 PM, Bryan Cantrill bryancantr...@gmail.com wrote: Yeah, that's busted, and I see the bug (we're calculating the size incorrectly); I'll fix that when I fix FIONREAD -- with my apologies... - Bryan On Wed, Dec 10, 2014 at 12:37 PM, Youzhong Yang youzh

[smartos-discuss] inotify panic

2014-12-16 Thread Youzhong Yang via smartos-discuss
Hello again, I tried to test samba with inotify under relatively heavy load, it panicked the machine. Does the stack give a hint about what's going wrong? I have the crash dump but I cannot upload it since it may contain some production data. If further analysis is needed, please advise.

Re: [smartos-discuss] inotify panic

2014-12-16 Thread Youzhong Yang via smartos-discuss
, is this reproducible? Could you reproduce it on a machine for which you could share the dump? Or perhaps tell me how you reproduced it? To answer your question: yes, further analysis is needed. ;) - Bryan On Tue, Dec 16, 2014 at 3:10 PM, Youzhong Yang via smartos-discuss smartos

Re: [smartos-discuss] inotify panic

2015-02-02 Thread Youzhong Yang via smartos-discuss
:56 AM, Youzhong Yang youzh...@gmail.com wrote: Hi Bryan, I managed to reproduce the issue using the attached C program. - compile: gcc -o inotify_stress inotify_stress.c - run: ./inotify_stress -f /var/log -n 120 If 120 does not cause crash, please use a larger number. It seems a stack