Yes tomcat is not starting up. I am also suspecting that EC2 instance was
probably compromised. Not sure as how but I see some rogue programs were
running under tomcat user. I use putty with private keys to login and those
keys are not in public view for sure.
These program were talking to some servers based out of China,Russia and
Germany with tcp,http and stratrum-tcp protocol with jsonp as data exchange
formt. I am not sure as how they got access to my ec2 instance and got
themselves installed.
I did some initial analysis on this one and have put those files in my g
drive which I have made public. I suspect either they have used tomcat to
gain access or they might have used yum updates for getting access to ec2
instance.
cronjobs.txt contains information that some programs were running with
tomcat user id.
hs_err_pid23773.log contains pid details for tomcat event.
jwzckuz.cf is probably some config file installed by hacker.
rciwd - was actual program which was consuming too much of swap and cpu and
was running as cron job. Not sure as what this is.
script.txt is actual script I extracted from one of the http request by
capturing traffic via wireshark.
files with names 0515 are tcpdump capture on the server taken while
unauthorised programs were running.
172-xx-68-244 is my ec2 instance and 98.122.xx.xx is my ip in the trace.
https://drive.google.com/drive/folders/1K5gfXTEvmuoIynCYtlwmf7DknyGkvhMI?usp=sharing
appreciate if someone from tomcat team take a look at all the files I have
attached in the drive.
Please let me know if more information is needed.
On Wed, May 16, 2018 at 11:09 AM, Christopher Schultz <
ch...@christopherschultz.net> wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Kiran,
>
> On 5/15/18 5:58 PM, Kiran Badi wrote:
> > For some reason my application hosted on ec2 is just not starting
> > up. I know I never had any memory issues in last 1 year or so.
> >
> > I see below trace in catalina.out file. I am not sure if I need to
> > add swap space or file permission is an issue here. Something
> > changed in ec2 that is causing this error.I think they auto updated
> > the tomcat version as well from 7.0.82 to 7.0.85.
> >
> > I seriously need some suggestions. I also need some suggestion as
> > how to prevent bots from trying to access manager app.
> >
> > May 14, 2018 8:44:46 PM org.apache.catalina.realm.LockOutRealm
> > filterLockedAccounts WARNING: An attempt was made to authenticate
> > the locked user "admin"
>
> It sure looks like Tomcat has started, since it is processing
> requests. Are you sure it's not starting?
>
> > java.util.logging.ErrorManager: 4 java.io.FileNotFoundException:
> > /usr/share/tomcat7/logs/catalina.2018-05-15.log (Permission
> > denied) at java.io.FileOutputStream.open0(Native Method)
>
> Obviously this is not memory-related. Did you intend to report this as
> a part of your problem?
> > Java HotSpot(TM) 64-Bit Server VM warning: INFO:
> > os::commit_memory(0x7f48f29d, 65536, 1) failed;
> > error='Cannot allocate memory' (errno=12) # # There is insufficient
> > memory for the Java Runtime Environment to continue. # Native
> > memory allocation (mmap) failed to map 65536 bytes for committing
> > reserved memory. # An error report file with more information is
> > saved as: # /usr/share/tomcat7/hs_err_pid23773.log # # Compiler
> > replay data is saved as:
> >
>
> The Java stack trace might be helpful, as would the native stack trace.
>
> What are your memory-related JVM launch parameters? What JVM are you
> using (version, architecture)?
>
> Odd that allocating 64kib should fail...
>
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlr8Sa0ACgkQHPApP6U8
> pFiBZg/+JmjmrlDUZuzoleg1ypwrxM51NSCCUPxLCxy/tI2UZF2MgRUwDZU3tdXX
> iHJsfwZ83bCt8m9eFBVy/4jWUQNjlDK+ahDBTOeqJvDkaNtdYLiLRBMqegtXF9JT
> cyt2nQdsetKx+rsI5HGytXBX6OuzJCSAw+bVHzzq2KFiOe4gnyqItsLg8TyXM+50
> giB0WlIBldyqj+kD9S8hRwqTTIXkAg4H+tI8+piBKKAojfLpuZB3qGhXhTncEMBA
> LL8Udbrz08vU3gXMg5U07pUHc/Vkn8U1axgcn4U3lQ0flKHRkBeabp/wVZ6a1Cuj
> a918715HRqZPezqEYoEYJjyUHV13c07T1nKFcLfR97VhFx1WjuTEGuHFriYjsPXN
> Qo0J6ej4+z0JItQVJ3w3qxijU9Vt0kEJq53raeclqNgdxhaVvLDDrPOxwZWvT9vz
> 1FiIyylRTNlC0tEAV3osQ9MFhf4eUgLGPGbEN69U+pEJ4Y2WgTlioKsueVDZcNrs
> czS6x0sR1Rd1waYQbnIXNpzIngQNAsnrw9cX73FSTmRVT3VGNdtlIFYzQ9aIl3UX
> 3cuLlqyumLySIV6BjORu6TgqGefSw+KYOJagTWo6IuExzLeU1vYs4V/ZVGt5qHQO
> kKLJmRaQozQ4u+ajMR9Lp5ESsLtjs+TPWy5tu4cQr6SE9PzL1fo=
> =Bm4c
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
