Hi,
Thanks for your thoughts.
What do you mean with an activeMQ pool? Can you point out a reference?
And why do you think that with only a few events an hour the process would
go to 100% immediately after startup?
We'll check out the load without kahaDb, but note that the worker/consumer
machine does not even have a kahaDb instance, so I can't see why it should
matter for the consumer/worker.
For the other issues:
- Java 1.7.0_03 was simply the most recent when we installed everything,
same with the activeMQ 5.6 version itsself. I'll switch to the latest
activeMQ version, I hope you remember a bug in our activeMQ version that
causes this.
- you're right about the multiples for xmx and xms, but we have had
multiples before as well and then also had 100% load. The broker runs
within several meg anyway but has a 100% load.
Here is the top on the worker/consumer with 12GB memory:
~top - 19:45:21 up 30 days, 3:26, 1 user, load average: 1.02, 1.04, 1.05
Tasks: 83 total, 1 running, 82 sleeping, 0 stopped, 0 zombie
Cpu(s): 17.0%us, 8.2%sy, 0.0%ni, 74.8%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 11899728k total, 725212k used, 11174516k free, 148112k buffers
Swap: 12056572k total, 0k used, 12056572k free, 320760k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
405 user 20 0 1736m 200m 10m S 100 1.7 214:55.31 java
829 zabbix 20 0 8680 1160 752 S 0 0.0 35:27.80 zabbix_agentd
830 zabbix 20 0 8680 1192 740 S 0 0.0 28:28.26 zabbix_agentd
833 zabbix 20 0 8680 1192 740 S 0 0.0 29:18.04 zabbix_agentd
1 root 20 0 3520 1836 1248 S 0 0.0 0:15.46 init
2 root 20 0 0 0 0 S 0 0.0 0:00.38 kthreadd
3 root 20 0 0 0 0 S 0 0.0 0:33.71 ksoftirqd/0
5 root 20 0 0 0 0 S 0 0.0 0:00.52 kworker/u:0
6 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0
7 root RT 0 0 0 0 S 0 0.0 0:07.38 watchdog/0
8 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1
9 root 20 0 0 0 0 S 0 0.0 0:10.29 kworker/1:0
10 root 20 0 0 0 0 S 0 0.0 0:02.46 ksoftirqd/1
12 root RT 0 0 0 0 S 0 0.0 0:05.08 watchdog/1
13 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/2
14 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/2:0
15 root 20 0 0 0 0 S 0 0.0 0:05.08 ksoftirqd/2
Thanks!
Niels
>
>
> So, here're a few of my observations and suggestions based on the provided
> info:
> 1. use a newer java than v1.7.0_03
> 2. switch to amq v5.7 or v5.8
> 3. keep xmx and xms the same and preferably a multiple of 512 megs
> 4. where are you getting server grade machines with 2 and 15 gig ram
> 5. when you notice 100% util, take a few spaced apart thread dumps and
> post them on here
> 6. on a 4 core machine, load should not just be 100% but higher than 1.
> can you post 'top' output as well
> 7. use the amq pool for producers, sessions, connections where possible
> 8. test your existing config and load without kahadb (non-persistent mode)
> and compare results
>
>
> On Apr 3, 2013, at 7:09, nielsbaloe <ni...@geoxplore.nl> wrote:
>
> Hi,
>
> -Xms512m -Xmx1500m
>
> but before we've set anything, it went wrong as well, and setting it
> higher also doesn't help. Note that there are only a few messages going
> through every hour, this is not yet fully production, so it should not
> even hit the 1% continuously...
>
> Thanks!,
> Niels
>
>>
>> Niels,
>>
>> I am not one of the experts here, but I am a new user of ActiveMQ via
>> TomEE, and I like to listen in on the mail topics/questions/responses
>> here.
>>
>> Since I listen in on Tomcat user list as well, I would say that this
>> sounds
>> like a GC (garbage collection) issue, but I might be mistaking.
>>
>> Can you reply with the java options of your app/container? What is your
>> java options for the Windows 7 laptops and the Ubuntu Linux
>> server/machine
>> that has 12GB and 2GB?
>>
>> I assume you have smaller memory settings on your Windows 7 (developer)
>> laptops and probably larger memory settings on the Ubuntu Linux
>> server/machine. Right? If yes, then GC may be the reason why you are
>> always
>> experiencing 100% CPU.
>>
>> This is just a guess/hunch, but since you provided such a detailed
>> question, please do not leave out the java (memory) options on your
>> Windows
>> 7 laptops as well as Ubuntu Linux server(s).
>>
>> Howard
>>
>>
>> On Wed, Apr 3, 2013 at 8:51 AM, nielsbaloe <ni...@geoxplore.nl> wrote:
>>
>>> Hi all,
>>>
>>> We are using activeMQ successfully for two projects now, but we
>>> accidentely
>>> discovered that both the broker and the worker/consumer machines are
>>> hitting
>>> 100% CPU continuously. We do not have this issue on our developers
>>> machines
>>> (all Windows 7 laptops). This occurs even when no events are being
>>> processed.
>>>
>>> I couldn't find any clues for this issue, except setting the prefetch
>>> size.
>>> I've set the prefetch size to 10, as we have 10 consumers at the
>>> worker/consumer machine. We have a broker machine and a worker/consumer
>>> machine, which are both configured like below. In the near future we
>>> will
>>> add more worker/consumer machines.
>>>
>>> OS: Ubuntu Linux 12.04 (headless)
>>> Memory: 12GB and 2GB
>>> CPU: Intel Xeon 3.06GHz 4core
>>> Java: "1.7.0_03", OpenJDK Runtime Environment (IcedTea7 2.1.1pre)
>>> (7~u3-2.1.1~pre1-1ubuntu3)
>>> Webcontainer: none, java-standalone
>>> ActiveMQ: 5.6.0
>>>
>>> The broker uses the internal KahaDB database.
>>>
>>> We are using one queue, to which the worker/consumer machine is
>>> listening
>>> and posting to, say about 100 messages a day. We also use about 4
>>> scheduled
>>> messages for every 'modem' (our internal subject) which results in
>>> about
>>> 40
>>> scheduled messages or so which generates an event once every 30 minuts.
>>> Nothing spectacular so to say.
>>>
>>> Thanks for any clues in advance. For completeness, I will post our
>>> Broker,
>>> Consumer and Producer code (without comments), this might show any
>>> wrong
>>> assumptions on our side.
>>>
>>> Best,
>>> Niels Baloe
>>>
>>>
>>>> -----------------------
>>>
>>> public class Consumer implements ExceptionListener {
>>>
>>> private Session session;
>>> private MessageConsumer messageConsumer;
>>> private static Logger LOG =
>>> Logger.getLogger(Consumer.class.getName());
>>>
>>> public Consumer(String brokerServer, String queueName,
>>> MessageListener messageListener) throws
>>> JMSException,
>>> FileNotFoundException, IOException {
>>> this(Broker.getSession(brokerServer), queueName,
>>> messageListener);
>>> }
>>>
>>> public Consumer(Session session, String queueName,
>>> MessageListener messageListener) throws
>>> JMSException {
>>> this.session = session;
>>>
>>> Queue queue = session.createQueue(queueName);
>>> messageConsumer = session.createConsumer(queue);
>>> messageConsumer.setMessageListener(messageListener);
>>> }
>>>
>>> public void close() {
>>> try {
>>> messageConsumer.close();
>>> } catch (JMSException e) {
>>> }
>>> try {
>>> session.close();
>>> } catch (JMSException e) {
>>> }
>>> }
>>>
>>> @Override
>>> public void onException(JMSException je) {
>>> LOG.log(Level.SEVERE, je.getMessage(), je);
>>> }
>>>
>>> }
>>>
>>> public class Producer {
>>>
>>> private Session session;
>>> private MessageProducer producer;
>>> public Producer(String brokerUrl, String queue) throws
>>> JMSException,
>>> FileNotFoundException, IOException {
>>> this(Broker.getSession(brokerUrl), queue);
>>> }
>>>
>>> public Producer(Session session, String queue) throws
>>> JMSException
>>> {
>>> this.session = session;
>>> Destination destination = session.createQueue(queue);
>>> producer = session.createProducer(destination);
>>> producer.setDeliveryMode(DeliveryMode.PERSISTENT);
>>> }
>>>
>>> public void close() {
>>> try {
>>> producer.close();
>>> } catch (JMSException e) {
>>> }
>>> try {
>>> session.close();
>>> } catch (JMSException je) {
>>> }
>>> }
>>>
>>> public Message getMessageText(String text) throws JMSException {
>>> return session.createTextMessage(text);
>>> }
>>>
>>> public Message getMessageObject() throws JMSException {
>>> return session.createObjectMessage();
>>> }
>>>
>>> public void send(Message message) throws JMSException {
>>> producer.send(message);
>>> }
>>>
>>> public void sendScheduled(Message message, String cron) throws
>>> JMSException
>>> {
>>>
>>> message.setStringProperty(ScheduledMessage.AMQ_SCHEDULED_CRON, cron);
>>> producer.send(message);
>>> }
>>>
>>> }
>>>
>>> public class Broker {
>>>
>>> private BrokerService broker;
>>>
>>> public Broker(String host, int port, String brokerName) throws
>>> Exception {
>>> broker = new BrokerService();
>>> broker.setUseJmx(true);
>>> broker.setBrokerName(brokerName);
>>> broker.addConnector("tcp://" + host + ":" + port);
>>> broker.setSchedulerSupport(true);
>>> broker.start();
>>> }
>>>
>>> public URI getNameTCP() {
>>> return broker.getVmConnectorURI();
>>> }
>>>
>>> public void close() {
>>> try {
>>> broker.stop();
>>> broker.waitUntilStopped();
>>> } catch (Exception e) {
>>> }
>>> }
>>>
>>> public static void closeConnection() {
>>> if (connection != null) {
>>> try {
>>> connection.close();
>>> } catch (JMSException e) {
>>> }
>>> }
>>> }
>>>
>>> private static Connection connection;
>>>
>>> private static Session getSessionWithoutRetry(String
>>> brokerServer)
>>> throws JMSException, FileNotFoundException,
>>> IOException {
>>> if (connection == null) { // does not work when broker
>>> is
>>> local
>>> ActiveMQConnectionFactory connectionFactory =
>>> new
>>> ActiveMQConnectionFactory(
>>> brokerServer);
>>> connectionFactory.setAlwaysSessionAsync(true);
>>>
>>> // Prefetch size
>>> String prefetch =
>>> NoImportUtils.getSettings().getProperty(
>>> "broker.prefetchSize");
>>>
>>> connectionFactory.getPrefetchPolicy().setAll(
>>> Integer.parseInt(prefetch));
>>> connection =
>>> connectionFactory.createConnection();
>>> connection.start();
>>> }
>>> return connection.createSession(false,
>>> Session.AUTO_ACKNOWLEDGE);
>>> }
>>>
>>> public static Session getSession(String brokerServer) throws
>>> JMSException,
>>> FileNotFoundException, IOException {
>>> try {
>>> return getSessionWithoutRetry(brokerServer);
>>> } catch (ConnectionFailedException e) {
>>> // Retry once when connection failed
>>> closeConnection();
>>> return getSessionWithoutRetry(brokerServer);
>>> }
>>> }
>>>
>>> }
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://activemq.2283324.n4.nabble.com/100-CPU-tp4665414.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>>
>>
>> _______________________________________________
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://activemq.2283324.n4.nabble.com/100-CPU-tp4665414p4665415.html
>>
>> To unsubscribe from 100% CPU, visit
>> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4665414&code=bmllbHNAZ2VveHBsb3JlLm5sfDQ2NjU0MTR8NTk0NzAwMTI4
>
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/100-CPU-tp4665414p4665420.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
>
> _______________________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://activemq.2283324.n4.nabble.com/100-CPU-tp4665414p4665431.html
>
> To unsubscribe from 100% CPU, visit
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4665414&code=bmllbHNAZ2VveHBsb3JlLm5sfDQ2NjU0MTR8NTk0NzAwMTI4
--
View this message in context:
http://activemq.2283324.n4.nabble.com/100-CPU-tp4665414p4665442.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
