Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Thank you very much! Terry; issues is fixed. I bumped that up Java heap
size to 512MB as well.
Thanks,
-Motty
On 06/24/2015 09:08 AM, Terry Siu wrote:
Hi Motty,
On your EventServer config, check what the java heap size setting is.
Ours was set to 128MB, which was much less than the default of 1GB.
After bumping up our value to 512MB, the EventServer started up fine
and no longer restarted. Did you have see any stderr logs for your
EventServer? The only ones I saw were from stdout and they did not
record any OOM. One of the Cloudera support folks said that I should
see OOM errors in the stderr logs, which is one cause of the
EventServer restart. Hope this helps.
-Terry
On Tuesday, June 23, 2015 at 10:07:29 AM UTC-7, motty cruz wrote:
yes, I did upgrade to 5.4.1:
*Version*: Cloudera Express 5.4.1 (#197 built by jenkins on
20150509-0041 git: 003e06d761f80834d39c3a42431a266f0aaee736)
I did not enable trace level as suggest!
Please let me know if you find a solution, I had tried increasing
memory to that service but no successful results yet.
Thanks,
-Motty
On 06/23/2015 09:31 AM, Terry Siu wrote:
Motty,
Did you upgrade your Cloudera Manager to 5.4.1? I was comparing
your INFO message for the EventCatcherService and noticed that
your Version number at the end says 5.3.1.
On Tuesday, June 23, 2015 at 9:26:42 AM UTC-7, Terry Siu wrote:
Bit skeptical that enabling trace would fix this issue, but
I did and no change. Event Server keeps restarting and no
errors. Posted a message on the Cloudera Manager forum to see
if anybody has any other ideas.
On Tuesday, June 23, 2015 at 9:01:08 AM UTC-7, Terry Siu wrote:
Hi Motty,
No, I haven't. The Event Server has been restarting
constantly since I left it last night and the logs do not
provide any errors on what might be wrong. Did you try
enabling the trace level as described by Atul? I'll see
if I can figure out where the setting is. This is getting
annoying and I'm going to engage Cloudera Support for this.
-Terry
On Tuesday, June 23, 2015 at 7:26:10 AM UTC-7, motty cruz
wrote:
Hello Terry,
did you find a solution to this problem? We're
experiencing this issue, no solution thus far.
Thanks,
-Motty
On 06/22/2015 03:12 PM, Terry Siu wrote:
Hi guys,
I just updated CDH from 5.3.3 to 5.4.2 and am seeing
the same issue with the Clouder Manager EventServer
(unexpected exits). I tailed the event server log
and saw not ERROR logs and it looked like the
EventServer keeps restarting itself over and over
again. The only notable thing I've seen in the log
is WARN level message:
2015-06-22 15:04:10,885 WARN
com.cloudera.cmf.event.publish.EventStorePublisherWithRetry:
F
ailed to publish event:
SimpleEvent{attributes={ROLE_TYPE=[EVENTSERVER],
CATEGORY=[LOG_MESS
AGE],
ROLE=[mgmt-EVENTSERVER-fd434ec7afa4142d2e2f91566bafb72b],
SEVERITY=[IMPORTANT], SERVI
CE=[mgmt],
HOST_IDS=[10a65ece-add7-4512-82bf-9f6db5c6c7ec],
SERVICE_TYPE=[MGMT], LOG_LEVEL=
[WARN], HOSTS=[8K04.corp.pivotlink.com
http://8K04.corp.pivotlink.com],
EVENTCODE=[EV_LOG_EVENT]}, content=Category SERVIC
E is not ROLE for input role type KMS-KMS.,
timestamp=1435010650675}
Anyone else got this resolved/found a workaround?
Thanks,
-Terry
On Thursday, June 4, 2015 at 7:24:16 AM UTC-7, motty
cruz wrote:
Thanks for your reply Atul,
I believe it has to do with version of Java. We
Also updated Java to version 8 previously we
were using java version 1.7-67.
I have not resolve the issue yet. do you mind
sharing the steps you taken to enable trace.
Thanks,
Motty
On 06/03/2015 06:18 PM, Atul Kulkarni wrote:
We had the same issue (upgrade from 5.3.3 to
5.4.0) - while trying to debug I enabled trace
and it started working correctly - it was very
odd, in the mean time we did not care about the
previous events data, hence we also whacked
directory (after backing it up, of course!)
where it stores some
Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Hello Terry,
did you find a solution to this problem? We're experiencing this issue,
no solution thus far.
Thanks,
-Motty
On 06/22/2015 03:12 PM, Terry Siu wrote:
Hi guys,
I just updated CDH from 5.3.3 to 5.4.2 and am seeing the same issue
with the Clouder Manager EventServer (unexpected exits). I tailed the
event server log and saw not ERROR logs and it looked like the
EventServer keeps restarting itself over and over again. The only
notable thing I've seen in the log is WARN level message:
2015-06-22 15:04:10,885 WARN
com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: F
ailed to publish event:
SimpleEvent{attributes={ROLE_TYPE=[EVENTSERVER], CATEGORY=[LOG_MESS
AGE], ROLE=[mgmt-EVENTSERVER-fd434ec7afa4142d2e2f91566bafb72b],
SEVERITY=[IMPORTANT], SERVI
CE=[mgmt], HOST_IDS=[10a65ece-add7-4512-82bf-9f6db5c6c7ec],
SERVICE_TYPE=[MGMT], LOG_LEVEL=
[WARN], HOSTS=[8K04.corp.pivotlink.com], EVENTCODE=[EV_LOG_EVENT]},
content=Category SERVIC
E is not ROLE for input role type KMS-KMS., timestamp=1435010650675}
Anyone else got this resolved/found a workaround?
Thanks,
-Terry
On Thursday, June 4, 2015 at 7:24:16 AM UTC-7, motty cruz wrote:
Thanks for your reply Atul,
I believe it has to do with version of Java. We Also updated Java
to version 8 previously we were using java version 1.7-67.
I have not resolve the issue yet. do you mind sharing the steps
you taken to enable trace.
Thanks,
Motty
On 06/03/2015 06:18 PM, Atul Kulkarni wrote:
We had the same issue (upgrade from 5.3.3 to 5.4.0) - while
trying to debug I enabled trace and it started working correctly
- it was very odd, in the mean time we did not care about the
previous events data, hence we also whacked directory (after
backing it up, of course!) where it stores some intermediate data.
As this is not a perfect or even sensible solution this may not
work for you, we are still investigating the real reason but
since we have not been able reproduce it - we are not sure how to
debug further.
Just thought of sharing this random act that we still can't
believe turned out to be the solution. Let us know if this works
for you too.
Regards,
Atul.
On Wednesday, June 3, 2015 at 10:30:13 AM UTC-7, motty cruz wrote:
Hello,
Just finished upgrading CDH from 5.3.1 to 5.4.1 using
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html
however Event health issues warnings eventserver
(cloudera1)
http://cloudera1.sscs.ad:7180/cmf/services/6/instances/28/status
Process Status, Unexpected Exits
2015-02-24 14:18:06,012 INFO
com.cloudera.cmf.eventcatcher.server.EventCatcherService:
Starting EventCatcherService. JVM Args:
[-XX:+UseConcMarkSweepGC, -XX:-CMSConcurrentMTEnabled,
-XX:+UseParNewGC,
-Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out,
-Djava.awt.headless=true, -Djava.net.preferIPv4Stack=true,
-Xms52428800, -Xmx52428800,
-XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh],
Args: [], Version: 5.3.1 (#191 built by jenkins on
20150123-2020 git: b0377087cf605a686591e659eb14078923bc3c83)
Any ideas? we're using Java 8
Thanks,
Motty
--
---
You received this message because you are subscribed to the
Google Groups CDH Users group.
To unsubscribe from this group and stop receiving emails from it,
send an email to cdh-user+u...@cloudera.org javascript:.
For more options, visit
https://groups.google.com/a/cloudera.org/d/optout
https://groups.google.com/a/cloudera.org/d/optout.
--
---
You received this message because you are subscribed to the Google
Groups CDH Users group.
To unsubscribe from this group and stop receiving emails from it, send
an email to cdh-user+unsubscr...@cloudera.org
mailto:cdh-user+unsubscr...@cloudera.org.
For more options, visit https://groups.google.com/a/cloudera.org/d/optout.
Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
yes, I did upgrade to 5.4.1:
*Version*: Cloudera Express 5.4.1 (#197 built by jenkins on
20150509-0041 git: 003e06d761f80834d39c3a42431a266f0aaee736)
I did not enable trace level as suggest!
Please let me know if you find a solution, I had tried increasing memory
to that service but no successful results yet.
Thanks,
-Motty
On 06/23/2015 09:31 AM, Terry Siu wrote:
Motty,
Did you upgrade your Cloudera Manager to 5.4.1? I was comparing your
INFO message for the EventCatcherService and noticed that your Version
number at the end says 5.3.1.
On Tuesday, June 23, 2015 at 9:26:42 AM UTC-7, Terry Siu wrote:
Bit skeptical that enabling trace would fix this issue, but I
did and no change. Event Server keeps restarting and no errors.
Posted a message on the Cloudera Manager forum to see if anybody
has any other ideas.
On Tuesday, June 23, 2015 at 9:01:08 AM UTC-7, Terry Siu wrote:
Hi Motty,
No, I haven't. The Event Server has been restarting constantly
since I left it last night and the logs do not provide any
errors on what might be wrong. Did you try enabling the trace
level as described by Atul? I'll see if I can figure out where
the setting is. This is getting annoying and I'm going to
engage Cloudera Support for this.
-Terry
On Tuesday, June 23, 2015 at 7:26:10 AM UTC-7, motty cruz wrote:
Hello Terry,
did you find a solution to this problem? We're
experiencing this issue, no solution thus far.
Thanks,
-Motty
On 06/22/2015 03:12 PM, Terry Siu wrote:
Hi guys,
I just updated CDH from 5.3.3 to 5.4.2 and am seeing the
same issue with the Clouder Manager EventServer
(unexpected exits). I tailed the event server log and saw
not ERROR logs and it looked like the EventServer keeps
restarting itself over and over again. The only notable
thing I've seen in the log is WARN level message:
2015-06-22 15:04:10,885 WARN
com.cloudera.cmf.event.publish.EventStorePublisherWithRetry:
F
ailed to publish event:
SimpleEvent{attributes={ROLE_TYPE=[EVENTSERVER],
CATEGORY=[LOG_MESS
AGE],
ROLE=[mgmt-EVENTSERVER-fd434ec7afa4142d2e2f91566bafb72b],
SEVERITY=[IMPORTANT], SERVI
CE=[mgmt],
HOST_IDS=[10a65ece-add7-4512-82bf-9f6db5c6c7ec],
SERVICE_TYPE=[MGMT], LOG_LEVEL=
[WARN], HOSTS=[8K04.corp.pivotlink.com
http://8K04.corp.pivotlink.com],
EVENTCODE=[EV_LOG_EVENT]}, content=Category SERVIC
E is not ROLE for input role type KMS-KMS.,
timestamp=1435010650675}
Anyone else got this resolved/found a workaround?
Thanks,
-Terry
On Thursday, June 4, 2015 at 7:24:16 AM UTC-7, motty cruz
wrote:
Thanks for your reply Atul,
I believe it has to do with version of Java. We Also
updated Java to version 8 previously we were using
java version 1.7-67.
I have not resolve the issue yet. do you mind sharing
the steps you taken to enable trace.
Thanks,
Motty
On 06/03/2015 06:18 PM, Atul Kulkarni wrote:
We had the same issue (upgrade from 5.3.3 to 5.4.0)
- while trying to debug I enabled trace and it
started working correctly - it was very odd, in the
mean time we did not care about the previous events
data, hence we also whacked directory (after backing
it up, of course!) where it stores some intermediate
data.
As this is not a perfect or even sensible solution
this may not work for you, we are still
investigating the real reason but since we have not
been able reproduce it - we are not sure how to
debug further.
Just thought of sharing this random act that we
still can't believe turned out to be the solution.
Let us know if this works for you too.
Regards,
Atul.
On Wednesday, June 3, 2015 at 10:30:13 AM UTC-7,
motty cruz wrote:
Hello,
Just finished upgrading CDH from 5.3.1 to 5.4.1
using
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html
however Event health issues warnings
Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Thanks for your reply Atul, I believe it has to do with version of Java. We Also updated Java to version 8 previously we were using java version 1.7-67. I have not resolve the issue yet. do you mind sharing the steps you taken to enable trace. Thanks, Motty On 06/03/2015 06:18 PM, Atul Kulkarni wrote: We had the same issue (upgrade from 5.3.3 to 5.4.0) - while trying to debug I enabled trace and it started working correctly - it was very odd, in the mean time we did not care about the previous events data, hence we also whacked directory (after backing it up, of course!) where it stores some intermediate data. As this is not a perfect or even sensible solution this may not work for you, we are still investigating the real reason but since we have not been able reproduce it - we are not sure how to debug further. Just thought of sharing this random act that we still can't believe turned out to be the solution. Let us know if this works for you too. Regards, Atul. On Wednesday, June 3, 2015 at 10:30:13 AM UTC-7, motty cruz wrote: Hello, Just finished upgrading CDH from 5.3.1 to 5.4.1 using http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html however Event health issues warnings eventserver (cloudera1) http://cloudera1.sscs.ad:7180/cmf/services/6/instances/28/status Process Status, Unexpected Exits 2015-02-24 14:18:06,012 INFO com.cloudera.cmf.eventcatcher.server.EventCatcherService: Starting EventCatcherService. JVM Args: [-XX:+UseConcMarkSweepGC, -XX:-CMSConcurrentMTEnabled, -XX:+UseParNewGC, -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out, -Djava.awt.headless=true, -Djava.net.preferIPv4Stack=true, -Xms52428800, -Xmx52428800, -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh], Args: [], Version: 5.3.1 (#191 built by jenkins on 20150123-2020 git: b0377087cf605a686591e659eb14078923bc3c83) Any ideas? we're using Java 8 Thanks, Motty -- --- You received this message because you are subscribed to the Google Groups CDH Users group. To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+unsubscr...@cloudera.org mailto:cdh-user+unsubscr...@cloudera.org. For more options, visit https://groups.google.com/a/cloudera.org/d/optout.
upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Hello, Just finished upgrading CDH from 5.3.1 to 5.4.1 using http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html however Event health issues warnings eventserver (cloudera1) http://cloudera1.sscs.ad:7180/cmf/services/6/instances/28/status Process Status, Unexpected Exits 2015-02-24 14:18:06,012 INFO com.cloudera.cmf.eventcatcher.server.EventCatcherService: Starting EventCatcherService. JVM Args: [-XX:+UseConcMarkSweepGC, -XX:-CMSConcurrentMTEnabled, -XX:+UseParNewGC, -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out, -Djava.awt.headless=true, -Djava.net.preferIPv4Stack=true, -Xms52428800, -Xmx52428800, -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh], Args: [], Version: 5.3.1 (#191 built by jenkins on 20150123-2020 git: b0377087cf605a686591e659eb14078923bc3c83) Any ideas? we're using Java 8 Thanks, Motty
