Thank you. This is a very useful find. Could you please explain where JAVA_TOOL_OPTIONS is set, will all userids see this variable ? In general I was under the impression that processes started through Trafodion monitor got their env settings from ms.env alone. For udrserv to see this setting, we may need to add it to ms.env (copy to all nodes then) and restart Trafdodion. Please do not do this till someone else confirms though as I am not certain.
Thanks Suresh On Fri, Sep 18, 2015 at 12:09 PM, Radu Marias <[email protected]> wrote: > ok, it seems that it didn't last long, I've got again the java heap issue: > > # java -version > Picked up JAVA_TOOL_OPTIONS: -Xms128m -Xmx2048m > Error occurred during initialization of VM > Could not reserve enough space for object heap > Error: Could not create the Java Virtual Machine. > Error: A fatal exception has occurred. Program will exit. > > Still no crash in trafodion after those 30 minutes. > > On Fri, Sep 18, 2015 at 8:06 PM, Radu Marias <[email protected]> wrote: > > > I think I managed to fix the java -version heap issue with: > > *export JAVA_TOOL_OPTIONS="-Xms128m -Xmx2048m"* > > For processes that will specify explicit values for Xms and Xmx those > will > > override the JAVA_TOO_OPTIONS as described here > > > http://stackoverflow.com/questions/28327620/difference-between-java-options-java-tool-options-and-java-opts > > > > trafodion processes will take these values if they don't specify others, > I > > see lines like this when trafodion starts: > > Picked up JAVA_TOOL_OPTIONS: -Xms128m -Xmx2048m > > > > But in hammerdb when creating stored procedures I get this error: > > Error in Virtual User 1: Picked up JAVA_TOOL_OPTIONS: -Xms128m -Xmx2048m > > > > FIxed the hammerdb issue by manually creating the SPJ and indexes but at > > first run with 5 user I've got the crash from TRAFODION-1492. After > > restarted trafodion managed to run with 5 users for about 30 minutes. > Will > > do more tests on Monday. > > > > On Fri, Sep 18, 2015 at 6:05 PM, Suresh Subbiah < > > [email protected]> wrote: > > > >> Thank you. > >> > >> Trafodion-1492 can be used to track both the esp crash and the udrserv > >> crash. The fixes will be in different areas, but that does not matter. > >> > >> I don't know much about containers or openVZ. Maybe others will know. > Hope > >> to have a fix ready soon for the udrserv crash problem, in case > container > >> settings cannot be changed. > >> > >> The general idea suggested by Selva is that we introduce env variables > >> with > >> min and max jvm heap size settings for the udrserv process (just like we > >> have today for executor processes). Udrserv does have the idea of > reading > >> from a configuration file, so we could use that approach if that is > >> preferable. Either way there should be some way to start a udrserv > process > >> with a smaller heap soon. > >> > >> Thanks > >> Suresh > >> > >> > >> > >> On Fri, Sep 18, 2015 at 8:17 AM, Radu Marias <[email protected]> > >> wrote: > >> > >> > The nodes are in OpenVZ containers an I noticed this: > >> > > >> > # cat /proc/user_beancounters > >> > uid resource held > maxheld > >> > barrier limit failcnt > >> > * privvmpages 6202505 9436485 > >> > 9437184 9437184 1573* > >> > > >> > I assume this could be related to java -version issue. Trying to see > if > >> I > >> > can fix this, we are limited on what can be set from inside the > >> container. > >> > > >> > # cat /proc/user_beancounters > >> > Version: 2.5 > >> > uid resource held maxheld > >> > barrier limit failcnt > >> > 10045785: kmemsize 111794747 999153664 > >> > 9223372036854775807 9223372036854775807 0 > >> > lockedpages 7970 7970 > >> > 6291456 6291456 0 > >> > * privvmpages 6202505 9436485 > >> > 9437184 9437184 1573* > >> > shmpages 34617 36553 > >> > 9223372036854775807 9223372036854775807 0 > >> > dummy 0 0 > >> > 9223372036854775807 9223372036854775807 0 > >> > numproc 952 1299 > >> > 30000 30000 0 > >> > physpages 1214672 6291456 > >> > 6291456 6291456 0 > >> > vmguarpages 0 0 > >> > 6291456 6291456 0 > >> > oomguarpages 1096587 2121834 > >> > 6291456 6291456 0 > >> > numtcpsock 226 457 > >> > 30000 30000 0 > >> > numflock 5 16 > >> > 1000 1100 0 > >> > numpty 4 6 > >> > 512 512 0 > >> > numsiginfo 1 69 > >> > 1024 1024 0 > >> > tcpsndbuf 5637456 17822864 > >> > 9223372036854775807 9223372036854775807 0 > >> > tcprcvbuf 6061504 13730792 > >> > 9223372036854775807 9223372036854775807 0 > >> > othersockbuf 46240 1268016 > >> > 9223372036854775807 9223372036854775807 0 > >> > dgramrcvbuf 0 436104 > >> > 9223372036854775807 9223372036854775807 0 > >> > numothersock 89 134 > >> > 30000 30000 0 > >> > dcachesize 61381173 935378121 > >> > 9223372036854775807 9223372036854775807 0 > >> > numfile 7852 11005 > >> > 250000 250000 0 > >> > dummy 0 0 > >> > 9223372036854775807 9223372036854775807 0 > >> > dummy 0 0 > >> > 9223372036854775807 9223372036854775807 0 > >> > dummy 0 0 > >> > 9223372036854775807 9223372036854775807 0 > >> > numiptent 38 38 > >> > 1000 1000 0 > >> > > >> > > >> > On Fri, Sep 18, 2015 at 3:34 PM, Radu Marias <[email protected]> > >> wrote: > >> > > >> > > With 1 user no crash occurs, but on the node on which hammerdb is > >> started > >> > > I noticed from time to time this: > >> > > > >> > > $ java -version > >> > > Error occurred during initialization of VM > >> > > Unable to allocate 199232KB bitmaps for parallel garbage collection > >> for > >> > > the requested 6375424KB heap. > >> > > Error: Could not create the Java Virtual Machine. > >> > > Error: A fatal exception has occurred. Program will exit. > >> > > > >> > > $ free -h > >> > > total used free shared buffers > >> cached > >> > > Mem: 24G 4.7G 19G 132M 0B > >> 314M > >> > > -/+ buffers/cache: 4.4G 19G > >> > > Swap: 0B 0B 0B > >> > > > >> > > > >> > > On Fri, Sep 18, 2015 at 2:21 PM, Radu Marias <[email protected]> > >> > wrote: > >> > > > >> > >> $ $JAVA_HOME/bin/java -XX:+PrintFlagsFinal -version | grep HeapSize > >> > >> uintx ErgoHeapSizeLimit = 0 > >> > >> {product} > >> > >> uintx HeapSizePerGCThread = 87241520 > >> > >> {product} > >> > >> uintx InitialHeapSize := 402653184 > >> > >> {product} > >> > >> uintx LargePageHeapSizeThreshold = 134217728 > >> > >> {product} > >> > >> uintx MaxHeapSize := 6442450944 > >> > >> {product} > >> > >> java version "1.7.0_67" > >> > >> Java(TM) SE Runtime Environment (build 1.7.0_67-b01) > >> > >> Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode) > >> > >> > >> > >> > >> > >> On Fri, Sep 18, 2015 at 2:20 PM, Radu Marias <[email protected] > > > >> > >> wrote: > >> > >> > >> > >>> I've logged this issue several days ago, is this ok? > >> > >>> https://issues.apache.org/jira/browse/TRAFODION-1492 > >> > >>> > >> > >>> Will try with one user and let you know. > >> > >>> > >> > >>> On Fri, Sep 18, 2015 at 7:05 AM, Suresh Subbiah < > >> > >>> [email protected]> wrote: > >> > >>> > >> > >>>> Hi > >> > >>>> > >> > >>>> How many Virtual users are being used? If it is more than one > >> could we > >> > >>>> please try the case with 1 user first. > >> > >>>> > >> > >>>> When the crash happens next time could we please try > >> > >>>> sqps | grep esp | wc -l > >> > >>>> > >> > >>>> If this number is large we know a lot of esp processes are being > >> > started > >> > >>>> which could consume memory. > >> > >>>> If this is the case please insert this row into the defaults > table > >> > from > >> > >>>> sqlci and the restart dcs (dcsstop followed by dcsstart) > >> > >>>> insert into "_MD_".defaults values('ATTEMPT_ESP_PARALLELISM', > >> 'OFF', > >> > >>>> 'hammerdb testing') ; > >> > >>>> exit ; > >> > >>>> > >> > >>>> I will work having the udr process create a JVM with a smaller > >> initial > >> > >>>> heap > >> > >>>> size. If you have time and would like to do so a, JIRA you file > >> will > >> > be > >> > >>>> helpful. Or I can file the JIRA and work on it. It will not take > >> long > >> > to > >> > >>>> make this change. > >> > >>>> > >> > >>>> Thanks > >> > >>>> Suresh > >> > >>>> > >> > >>>> PS I found this command from stackOverflow to determine the > >> > >>>> initialHeapSize > >> > >>>> we get by default in this env > >> > >>>> > >> > >>>> java -XX:+PrintFlagsFinal -version | grep HeapSize > >> > >>>> > >> > >>>> > >> > >>>> > >> > >>>> > >> > >>>> > >> > > >> > http://stackoverflow.com/questions/4667483/how-is-the-default-java-heap-size-determined > >> > >>>> > >> > >>>> > >> > >>>> > >> > >>>> On Thu, Sep 17, 2015 at 10:32 AM, Radu Marias < > >> [email protected]> > >> > >>>> wrote: > >> > >>>> > >> > >>>> > Did the steps mentioned above to ensure that the trafodion > >> processes > >> > >>>> are > >> > >>>> > free of JAVA installation mixup. > >> > >>>> > Also changed so that hdp, trafodion and hammerdb uses the same > >> jdk > >> > >>>> from > >> > >>>> > */usr/jdk64/jdk1.7.0_67* > >> > >>>> > > >> > >>>> > # java -version > >> > >>>> > java version "1.7.0_67" > >> > >>>> > Java(TM) SE Runtime Environment (build 1.7.0_67-b01) > >> > >>>> > Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode) > >> > >>>> > > >> > >>>> > # echo $JAVA_HOME > >> > >>>> > /usr/jdk64/jdk1.7.0_67 > >> > >>>> > > >> > >>>> > But when running hammerdb I got again a crash on 2 nodes. I > >> noticed > >> > >>>> that > >> > >>>> > before the crash for about one minute I'm getting errors for > >> *java > >> > >>>> > -version* and > >> > >>>> > about 30 seconds after the crash the java -version worked > again. > >> So > >> > >>>> these > >> > >>>> > issues might be related. Didn't yet found the problem and how > to > >> fix > >> > >>>> the > >> > >>>> > java -version issue. > >> > >>>> > > >> > >>>> > # java -version > >> > >>>> > Error occurred during initialization of VM > >> > >>>> > Could not reserve enough space for object heap > >> > >>>> > Error: Could not create the Java Virtual Machine. > >> > >>>> > Error: A fatal exception has occurred. Program will exit > >> > >>>> > > >> > >>>> > # file core.5813 > >> > >>>> > core.5813: ELF 64-bit LSB core file x86-64, version 1 (SYSV), > >> > >>>> SVR4-style, > >> > >>>> > from 'tdm_udrserv SQMON1.1 00000 00000 005813 $Z0004R3 > >> > >>>> > 188.138.61.175:48357 > >> > >>>> > 00004 000' > >> > >>>> > > >> > >>>> > #0 0x00007f6920ba0625 in raise () from /lib64/libc.so.6 > >> > >>>> > #1 0x00007f6920ba1e05 in abort () from /lib64/libc.so.6 > >> > >>>> > #2 0x0000000000424369 in comTFDS (msg1=0x43c070 "Trafodion UDR > >> > Server > >> > >>>> > Internal Error", msg2=<value optimized out>, > msg3=0x7fff119787f0 > >> > >>>> "Source > >> > >>>> > file information unavailable", > >> > >>>> > msg4=0x7fff11977ff0 "User routine being processed : > >> > >>>> > TRAFODION.TPCC.NEWORDER, Routine Type : Stored Procedure, > >> Language > >> > >>>> Type : > >> > >>>> > JAVA, Error occurred outside the user routine code", > >> msg5=0x43ddc3 > >> > "", > >> > >>>> > dialOut=<value optimized out>, writeToSeaLog=1) at > >> > >>>> > ../udrserv/UdrFFDC.cpp:191 > >> > >>>> > #3 0x00000000004245d7 in makeTFDSCall (msg=0x7f692324b310 "The > >> Java > >> > >>>> > virtual machine aborted", file=<value optimized out>, > line=<value > >> > >>>> optimized > >> > >>>> > out>, dialOut=1, writeToSeaLog=1) at ../udrserv/UdrFFDC.cpp:219 > >> > >>>> > #4 0x00007f69232316b8 in LmJavaHooks::abortHookJVM () at > >> > >>>> > ../langman/LmJavaHooks.cpp:54 > >> > >>>> > #5 0x00007f69229cbbc6 in ParallelScavengeHeap::initialize() () > >> from > >> > >>>> > /usr/jdk64/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so > >> > >>>> > #6 0x00007f6922afedba in Universe::initialize_heap() () from > >> > >>>> > /usr/jdk64/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so > >> > >>>> > #7 0x00007f6922afff89 in universe_init() () from > >> > >>>> > /usr/jdk64/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so > >> > >>>> > #8 0x00007f692273d9f5 in init_globals() () from > >> > >>>> > /usr/jdk64/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so > >> > >>>> > #9 0x00007f6922ae78ed in Threads::create_vm(JavaVMInitArgs*, > >> bool*) > >> > >>>> () > >> > >>>> > from /usr/jdk64/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so > >> > >>>> > #10 0x00007f69227c5a34 in JNI_CreateJavaVM () from > >> > >>>> > /usr/jdk64/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so > >> > >>>> > #11 0x00007f692322de51 in LmLanguageManagerJava::initialize > >> > >>>> (this=<value > >> > >>>> > optimized out>, result=<value optimized out>, maxLMJava=<value > >> > >>>> optimized > >> > >>>> > out>, userOptions=0x7f69239ba418, diagsArea=<value optimized > >> out>) > >> > at > >> > >>>> > ../langman/LmLangManagerJava.cpp:379 > >> > >>>> > #12 0x00007f692322f564 in > >> > LmLanguageManagerJava::LmLanguageManagerJava > >> > >>>> > (this=0x7f69239bec38, result=@0x7fff1197e19c, > >> commandLineMode=<value > >> > >>>> > optimized out>, maxLMJava=1, userOptions=0x7f69239ba418, > >> > >>>> > diagsArea=0x7f6923991780) at > ../langman/LmLangManagerJava.cpp:155 > >> > >>>> > #13 0x0000000000425619 in UdrGlobals::getOrCreateJavaLM > >> > >>>> > (this=0x7f69239ba040, result=@0x7fff1197e19c, diags=<value > >> optimized > >> > >>>> out>) > >> > >>>> > at ../udrserv/udrglobals.cpp:322 > >> > >>>> > #14 0x0000000000427328 in processALoadMessage > >> > (UdrGlob=0x7f69239ba040, > >> > >>>> > msgStream=..., request=..., env=<value optimized out>) at > >> > >>>> > ../udrserv/udrload.cpp:163 > >> > >>>> > #15 0x000000000042fbfd in processARequest > >> (UdrGlob=0x7f69239ba040, > >> > >>>> > msgStream=..., env=...) at ../udrserv/udrserv.cpp:660 > >> > >>>> > #16 0x000000000043269c in runServer (argc=2, > >> argv=0x7fff1197e528) at > >> > >>>> > ../udrserv/udrserv.cpp:520 > >> > >>>> > #17 0x000000000043294e in main (argc=2, argv=0x7fff1197e528) at > >> > >>>> > ../udrserv/udrserv.cpp:356 > >> > >>>> > > >> > >>>> > On Wed, Sep 16, 2015 at 6:03 PM, Suresh Subbiah < > >> > >>>> > [email protected]> > >> > >>>> > wrote: > >> > >>>> > > >> > >>>> > > Hi, > >> > >>>> > > > >> > >>>> > > I have added a wiki page that describes how to get a stack > >> trace > >> > >>>> from a > >> > >>>> > > core file. The page could do with some improvements on > finding > >> the > >> > >>>> core > >> > >>>> > > file and maybe even doing more than getting thestack trace. > For > >> > now > >> > >>>> it > >> > >>>> > > should make our troubleshooting cycle faster if the stack > >> trace is > >> > >>>> > included > >> > >>>> > > in the initial message itself. > >> > >>>> > > > >> > >>>> > > > >> > >>>> > > >> > >>>> > >> > > >> > https://cwiki.apache.org/confluence/display/TRAFODION/Obtain+stack+trace+from+a+core+file > >> > >>>> > > > >> > >>>> > > In this case, the last node does not seem to have gdb, so I > >> could > >> > >>>> not see > >> > >>>> > > the trace there. I moved the core file to the first node but > >> then > >> > >>>> the > >> > >>>> > trace > >> > >>>> > > looks like this. I assume this is because I moved the core > file > >> > to a > >> > >>>> > > different node. I think Selva's suggestion is good to try. We > >> may > >> > >>>> have > >> > >>>> > had > >> > >>>> > > a few tdm_udrserv processes from before the time the java > >> change > >> > >>>> was > >> > >>>> > made. > >> > >>>> > > > >> > >>>> > > $ gdb tdm_udrserv core.49256 > >> > >>>> > > #0 0x00007fe187a674fe in __longjmp () from /lib64/libc.so.6 > >> > >>>> > > #1 0x8857780a58ff2155 in ?? () > >> > >>>> > > Cannot access memory at address 0x8857780a58ff2155 > >> > >>>> > > > >> > >>>> > > The back trace we saw yesterday when a udrserv process exited > >> when > >> > >>>> JVM > >> > >>>> > > could not be started is used in the wiki page instead of this > >> one. > >> > >>>> If you > >> > >>>> > > have time a JIRA on this unexpected udrserv exit will also be > >> > >>>> valuable > >> > >>>> > for > >> > >>>> > > the Trafodion team. > >> > >>>> > > > >> > >>>> > > Thanks > >> > >>>> > > Suresh > >> > >>>> > > > >> > >>>> > > On Wed, Sep 16, 2015 at 8:39 AM, Selva Govindarajan < > >> > >>>> > > [email protected]> wrote: > >> > >>>> > > > >> > >>>> > > > Thanks for creating the JIRA Trafodion-1492. The error is > >> > >>>> similar to > >> > >>>> > > > scenario-2. The process tdm_udrserv dumped core. We will > look > >> > >>>> into the > >> > >>>> > > core > >> > >>>> > > > file. In the meantime, can you please do the following: > >> > >>>> > > > > >> > >>>> > > > Bring the Trafodion instance down > >> > >>>> > > > echo $MY_SQROOT -- shows Trafodion installation directory > >> > >>>> > > > Remove $MY_SQROOT/etc/ms.env from all nodes > >> > >>>> > > > > >> > >>>> > > > > >> > >>>> > > > Start a New Terminal Session so that new Java settings are > in > >> > >>>> place > >> > >>>> > > > Login as a Trafodion user > >> > >>>> > > > cd <trafodion_installation_directory> > >> > >>>> > > > . ./sqenv.sh (skip this if it is done automatically upon > >> logon) > >> > >>>> > > > sqgen > >> > >>>> > > > > >> > >>>> > > > Exit and Start a New Terminal Session > >> > >>>> > > > Restart the Trafodion instance and check if you are seeing > >> the > >> > >>>> issue > >> > >>>> > with > >> > >>>> > > > tdm_udrserv again. We wanted to ensure that the trafodion > >> > >>>> processes are > >> > >>>> > > > free > >> > >>>> > > > of JAVA installation mixup in your earlier message. We > >> suspect > >> > >>>> that can > >> > >>>> > > > cause tdm_udrserv process to dump core. > >> > >>>> > > > > >> > >>>> > > > > >> > >>>> > > > Selva > >> > >>>> > > > > >> > >>>> > > > -----Original Message----- > >> > >>>> > > > From: Radu Marias [mailto:[email protected]] > >> > >>>> > > > Sent: Wednesday, September 16, 2015 5:40 AM > >> > >>>> > > > To: dev <[email protected]> > >> > >>>> > > > Subject: Re: odbc and/or hammerdb logs > >> > >>>> > > > > >> > >>>> > > > I'm seeing this in hammerdb logs, I assume is due to the > >> crash > >> > >>>> and some > >> > >>>> > > > processes are stopped: > >> > >>>> > > > > >> > >>>> > > > Error in Virtual User 1: [Trafodion ODBC Driver][Trafodion > >> > >>>> Database] > >> > >>>> > SQL > >> > >>>> > > > ERROR:*** ERROR[2034] $Z0106BZ:16: Operating system error > 201 > >> > >>>> while > >> > >>>> > > > communicating with server process $Z010LPE:23. [2015-09-16 > >> > >>>> 12:35:33] > >> > >>>> > > > [Trafodion ODBC Driver][Trafodion Database] SQL ERROR:*** > >> > >>>> ERROR[8904] > >> > >>>> > SQL > >> > >>>> > > > did not receive a reply from MXUDR, possibly caused by > >> internal > >> > >>>> errors > >> > >>>> > > when > >> > >>>> > > > executing user-defined routines. [2015-09-16 12:35:33] > >> > >>>> > > > > >> > >>>> > > > $ sqcheck > >> > >>>> > > > Checking if processes are up. > >> > >>>> > > > Checking attempt: 1; user specified max: 2. Execution time > in > >> > >>>> seconds: > >> > >>>> > 0. > >> > >>>> > > > > >> > >>>> > > > The SQ environment is up! > >> > >>>> > > > > >> > >>>> > > > > >> > >>>> > > > Process Configured Actual Down > >> > >>>> > > > ------- ---------- ------ ---- > >> > >>>> > > > DTM 5 5 > >> > >>>> > > > RMS 10 10 > >> > >>>> > > > MXOSRVR 20 20 > >> > >>>> > > > > >> > >>>> > > > On Wed, Sep 16, 2015 at 3:28 PM, Radu Marias < > >> > >>>> [email protected]> > >> > >>>> > > wrote: > >> > >>>> > > > > >> > >>>> > > > > I've restarted hdp and trafodion and now I managed to > >> create > >> > the > >> > >>>> > > > > schema and stored procedures from hammerdb. But I'm > getting > >> > >>>> fails and > >> > >>>> > > > > dump core again by trafodion while running virtual users. > >> For > >> > >>>> some of > >> > >>>> > > > > the users I sometimes see in hammerdb logs: > >> > >>>> > > > > Vuser 5:Failed to execute payment > >> > >>>> > > > > Vuser 5:Failed to execute stock level > >> > >>>> > > > > Vuser 5:Failed to execute new order > >> > >>>> > > > > > >> > >>>> > > > > Core files are on out last node, feel free to examine > them, > >> > the > >> > >>>> files > >> > >>>> > > > > were dumped while getting hammerdb errors: > >> > >>>> > > > > > >> > >>>> > > > > *core.49256* > >> > >>>> > > > > > >> > >>>> > > > > *core.48633* > >> > >>>> > > > > > >> > >>>> > > > > *core.49290* > >> > >>>> > > > > > >> > >>>> > > > > > >> > >>>> > > > > On Wed, Sep 16, 2015 at 3:24 PM, Radu Marias < > >> > >>>> [email protected]> > >> > >>>> > > > wrote: > >> > >>>> > > > > > >> > >>>> > > > >> *Scenario 1:* > >> > >>>> > > > >> > >> > >>>> > > > >> I've created this issue > >> > >>>> > > > >> https://issues.apache.org/jira/browse/TRAFODION-1492 > >> > >>>> > > > >> I think another fix was made related to *Committed_AS* > in > >> > >>>> > > > >> *sql/cli/memmonitor.cpp*. > >> > >>>> > > > >> > >> > >>>> > > > >> This is a response from Narendra in a previous thread > >> where > >> > the > >> > >>>> > issue > >> > >>>> > > > >> was fixed to start the trafodion: > >> > >>>> > > > >> > >> > >>>> > > > >> > >> > >>>> > > > >>> > >> > >>>> > > > >>> > >> > >>>> > > > >>> > >> > >>>> > > > >>> *I updated the code: sql/cli/memmonitor.cpp, so that if > >> > >>>> > > > >>> /proc/meminfo does not have the ‘Committed_AS’ entry, > it > >> > will > >> > >>>> > ignore > >> > >>>> > > > >>> it. Built it and put the binary: libcli.so on the > >> veracity > >> > >>>> box (in > >> > >>>> > > > >>> the $MY_SQROOT/export/lib64 directory – on all the > >> nodes). > >> > >>>> > Restarted > >> > >>>> > > > the > >> > >>>> > > > >>> env and ‘sqlci’ worked fine. > >> > >>>> > > > >>> Was able to ‘initialize trafodion’ and create a table.* > >> > >>>> > > > >> > >> > >>>> > > > >> > >> > >>>> > > > >> *Scenario 2:* > >> > >>>> > > > >> > >> > >>>> > > > >> The *java -version* problem I recall we had only on the > >> other > >> > >>>> > cluster > >> > >>>> > > > >> with centos 7, I did't seen it on this one with centos > >> 6.7. > >> > >>>> But a > >> > >>>> > > > >> change I made these days in the latter one is installing > >> > >>>> oracle *jdk > >> > >>>> > > > >> 1.7.0_79* as default one and is where *JAVA_HOME* points > >> to. > >> > >>>> Before > >> > >>>> > > > >> that some nodes had *open-jdk* as default and others > >> didn't > >> > >>>> have one > >> > >>>> > > > >> but just the one installed by path by *ambari* in > >> > >>>> > > > >> */usr/jdk64/jdk1.7.0_67* but which was not linked to > >> > JAVA_HOME > >> > >>>> or > >> > >>>> > > *java* > >> > >>>> > > > >> command by *alternatives*. > >> > >>>> > > > >> > >> > >>>> > > > >> *Failures is HammerDB:* > >> > >>>> > > > >> > >> > >>>> > > > >> Attached is the *trafodion.dtm.**log* from a node on > >> which I > >> > >>>> see a > >> > >>>> > > > >> lot of lines like these and I assume is the *transaction > >> > >>>> conflict* > >> > >>>> > > > >> that you mentioned, I see these line on 4 out of 5 > nodes: > >> > >>>> > > > >> > >> > >>>> > > > >> 2015-09-14 12:21:49,413 INFO dtm.HBaseTxClient: > >> useForgotten > >> > >>>> is true > >> > >>>> > > > >> 2015-09-14 12:21:49,414 INFO dtm.HBaseTxClient: > >> > forceForgotten > >> > >>>> is > >> > >>>> > > > >> false > >> > >>>> > > > >> 2015-09-14 12:21:49,446 INFO dtm.TmAuditTlog: > >> > >>>> forceControlPoint is > >> > >>>> > > > >> false > >> > >>>> > > > >> 2015-09-14 12:21:49,446 INFO dtm.TmAuditTlog: > >> useAutoFlush is > >> > >>>> false > >> > >>>> > > > >> 2015-09-14 12:21:49,447 INFO dtm.TmAuditTlog: > >> ageCommitted is > >> > >>>> false > >> > >>>> > > > >> 2015-09-14 12:21:49,447 INFO dtm.TmAuditTlog: > >> > >>>> disableBlockCache is > >> > >>>> > > > >> false > >> > >>>> > > > >> 2015-09-14 12:21:52,229 INFO dtm.HBaseAuditControlPoint: > >> > >>>> > > > >> disableBlockCache is false > >> > >>>> > > > >> 2015-09-14 12:21:52,233 INFO dtm.HBaseAuditControlPoint: > >> > >>>> > useAutoFlush > >> > >>>> > > > >> is false > >> > >>>> > > > >> 2015-09-14 12:42:57,346 INFO dtm.HBaseTxClient: Exit > >> > >>>> RET_HASCONFLICT > >> > >>>> > > > >> prepareCommit, txid: 17179989222 > >> > >>>> > > > >> 2015-09-14 12:43:46,102 INFO dtm.HBaseTxClient: Exit > >> > >>>> RET_HASCONFLICT > >> > >>>> > > > >> prepareCommit, txid: 17179989277 > >> > >>>> > > > >> 2015-09-14 12:44:11,598 INFO dtm.HBaseTxClient: Exit > >> > >>>> RET_HASCONFLICT > >> > >>>> > > > >> prepareCommit, txid: 17179989309 > >> > >>>> > > > >> > >> > >>>> > > > >> What *transaction conflict* means in this case? > >> > >>>> > > > >> > >> > >>>> > > > >> On Wed, Sep 16, 2015 at 2:43 AM, Selva Govindarajan < > >> > >>>> > > > >> [email protected]> wrote: > >> > >>>> > > > >> > >> > >>>> > > > >>> Hi Radu, > >> > >>>> > > > >>> > >> > >>>> > > > >>> Thanks for using Trafodion. With the help from Suresh, > we > >> > >>>> looked at > >> > >>>> > > > >>> the core files in your cluster. We believe that there > are > >> > two > >> > >>>> > > > >>> scenarios that is causing the Trafodion processes to > dump > >> > >>>> core. > >> > >>>> > > > >>> > >> > >>>> > > > >>> Scenario 1: > >> > >>>> > > > >>> Core dumped by tdm_arkesp processes. Trafodion engine > has > >> > >>>> assumed > >> > >>>> > > > >>> the entity /proc/meminfo/Committed_AS is available in > all > >> > >>>> flavors > >> > >>>> > of > >> > >>>> > > > >>> linux. The absence of this entity is not handled > >> correctly > >> > >>>> by the > >> > >>>> > > > >>> trafodion tdm_arkesp process and hence it dumped core. > >> > Please > >> > >>>> file > >> > >>>> > a > >> > >>>> > > > >>> JIRA using this link > >> > >>>> > > > >>> > >> > >>>> https://issues.apache.org/jira/secure/CreateIssue!default.jspa > and > >> > >>>> > > > >>> choose "Apache Trafodion" as the project to report a > bug > >> > >>>> against. > >> > >>>> > > > >>> > >> > >>>> > > > >>> Scenario 2: > >> > >>>> > > > >>> Core dumped by tdm_udrserv processes. From our > analysis, > >> > this > >> > >>>> > > > >>> problem happened when the process attempted to create > the > >> > JVM > >> > >>>> > > > >>> instance programmatically. Few days earlier, we have > >> > observed > >> > >>>> > > > >>> similar issue in your cluster when java -version > command > >> was > >> > >>>> > > > >>> attempted. But, java -version or $JAVA_HOME/bin/java > >> > -version > >> > >>>> works > >> > >>>> > > > >>> fine now. > >> > >>>> > > > >>> Was there any change made to the cluster recently to > >> avoid > >> > the > >> > >>>> > > > >>> problem with java -version command? > >> > >>>> > > > >>> > >> > >>>> > > > >>> You can please delete all the core files in sql/scripts > >> > >>>> directory > >> > >>>> > > > >>> and issue the command to invoke SPJ and check if it > still > >> > >>>> dumps > >> > >>>> > > > >>> core. We can look at the core file if it happens again. > >> Your > >> > >>>> > > > >>> solution to the java -version command would be helpful. > >> > >>>> > > > >>> > >> > >>>> > > > >>> For the failures with HammerDB, can you please send us > >> the > >> > >>>> exact > >> > >>>> > > > >>> error message returned by the Trafodion engine to the > >> > >>>> application. > >> > >>>> > > > >>> This might help us to narrow down the cause. You can > also > >> > >>>> look at > >> > >>>> > > > >>> $MY_SQROOT/logs/trafodion.dtm.log to check if any > >> > transaction > >> > >>>> > > > >>> conflict is causing this error. > >> > >>>> > > > >>> > >> > >>>> > > > >>> Selva > >> > >>>> > > > >>> -----Original Message----- > >> > >>>> > > > >>> From: Radu Marias [mailto:[email protected]] > >> > >>>> > > > >>> Sent: Tuesday, September 15, 2015 9:09 AM > >> > >>>> > > > >>> To: dev <[email protected]> > >> > >>>> > > > >>> Subject: Re: odbc and/or hammerdb logs > >> > >>>> > > > >>> > >> > >>>> > > > >>> Also noticed there are several core. files from today > in > >> > >>>> > > > >>> */home/trafodion/trafodion-20150828_0830/sql/scripts*. > If > >> > >>>> needed > >> > >>>> > > > >>> please provide a gmail address so I can share them via > >> > gdrive. > >> > >>>> > > > >>> > >> > >>>> > > > >>> On Tue, Sep 15, 2015 at 6:29 PM, Radu Marias < > >> > >>>> [email protected] > >> > >>>> > > > >> > >>>> > > > >>> wrote: > >> > >>>> > > > >>> > >> > >>>> > > > >>> > Hi, > >> > >>>> > > > >>> > > >> > >>>> > > > >>> > I'm running HammerDB over trafodion and when running > >> > virtual > >> > >>>> > users > >> > >>>> > > > >>> > sometimes I get errors like this in hammerdb logs: > >> > >>>> > > > >>> > *Vuser 1:Failed to execute payment* > >> > >>>> > > > >>> > > >> > >>>> > > > >>> > *Vuser 1:Failed to execute new order* > >> > >>>> > > > >>> > > >> > >>>> > > > >>> > I'm using unixODBC and I tried to add these line in > >> > >>>> > > > >>> > */etc/odbc.ini* but the trace file is not created. > >> > >>>> > > > >>> > *[ODBC]* > >> > >>>> > > > >>> > *Trace = 1* > >> > >>>> > > > >>> > *TraceFile = /var/log/odbc_tracefile.log* > >> > >>>> > > > >>> > > >> > >>>> > > > >>> > Also tried with *Trace = yes* and *Trace = on*, I've > >> found > >> > >>>> > > > >>> > multiple references for both. > >> > >>>> > > > >>> > > >> > >>>> > > > >>> > How can I see more logs to debug the issue? Can I > >> enable > >> > >>>> logs for > >> > >>>> > > > >>> > all queries in trafodion? > >> > >>>> > > > >>> > > >> > >>>> > > > >>> > -- > >> > >>>> > > > >>> > And in the end, it's not the years in your life that > >> > count. > >> > >>>> It's > >> > >>>> > > > >>> > the life in your years. > >> > >>>> > > > >>> > > >> > >>>> > > > >>> > >> > >>>> > > > >>> > >> > >>>> > > > >>> > >> > >>>> > > > >>> -- > >> > >>>> > > > >>> And in the end, it's not the years in your life that > >> count. > >> > >>>> It's > >> > >>>> > the > >> > >>>> > > > >>> life in your years. > >> > >>>> > > > >>> > >> > >>>> > > > >> > >> > >>>> > > > >> > >> > >>>> > > > >> > >> > >>>> > > > >> -- > >> > >>>> > > > >> And in the end, it's not the years in your life that > >> count. > >> > >>>> It's the > >> > >>>> > > > life > >> > >>>> > > > >> in your years. > >> > >>>> > > > >> > >> > >>>> > > > > > >> > >>>> > > > > > >> > >>>> > > > > > >> > >>>> > > > > -- > >> > >>>> > > > > And in the end, it's not the years in your life that > count. > >> > >>>> It's the > >> > >>>> > > life > >> > >>>> > > > > in your years. > >> > >>>> > > > > > >> > >>>> > > > > >> > >>>> > > > > >> > >>>> > > > > >> > >>>> > > > -- > >> > >>>> > > > And in the end, it's not the years in your life that count. > >> It's > >> > >>>> the > >> > >>>> > life > >> > >>>> > > > in your years. > >> > >>>> > > > > >> > >>>> > > > >> > >>>> > > >> > >>>> > > >> > >>>> > > >> > >>>> > -- > >> > >>>> > And in the end, it's not the years in your life that count. > It's > >> the > >> > >>>> life > >> > >>>> > in your years. > >> > >>>> > > >> > >>>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> -- > >> > >>> And in the end, it's not the years in your life that count. It's > the > >> > >>> life in your years. > >> > >>> > >> > >> > >> > >> > >> > >> > >> > >> -- > >> > >> And in the end, it's not the years in your life that count. It's > the > >> > life > >> > >> in your years. > >> > >> > >> > > > >> > > > >> > > > >> > > -- > >> > > And in the end, it's not the years in your life that count. It's the > >> life > >> > > in your years. > >> > > > >> > > >> > > >> > > >> > -- > >> > And in the end, it's not the years in your life that count. It's the > >> life > >> > in your years. > >> > > >> > > > > > > > > -- > > And in the end, it's not the years in your life that count. It's the life > > in your years. > > > > > > -- > And in the end, it's not the years in your life that count. It's the life > in your years. >
