John, you can use the broker shipped with UIMA-AS installation for your services.
Jerry On Wed, Jul 19, 2017 at 10:55 AM, Osborne, John David (Campus) < [email protected]> wrote: > Jerry, my services would span across multiple jobs so a service model is > preferable for me. > > With regards to a secondary broker - I'm thinking to use a UIMA-AS > installation on the same machine rather than download the full Apache AMQ > runtime standalone. I'm worried about potential compatibility issues > between DUCC broker and downloaded broker (unfounded concern?) and I am > unfamiliar with AMQ in isolation. It should also allow for comparison > between AS and DUCC. Is this a reasonable approach? > > Thanks for all your help so far, > > -John > > > -----Original Message----- > From: [email protected] [mailto:[email protected]] On Behalf Of Jaroslaw > Cwiklik > Sent: Tuesday, July 18, 2017 3:40 PM > To: [email protected] > Subject: Re: DUCC Security Model for Service Deployment > > John, services are typically long running and independent of jobs. They > provide a specific function and may take a long time to initialize. They > are meant to be reusable across multiple jobs. If your analytics are fast > initializing and not meant to be shared you can just use jobs. > > We should probably modify the documentation to talk about recommending a > secondary broker for services. My recommendation is to download and install > the full AMQ runtime for your services. DUCC only bundles essential parts > of AMQ distro. > > You decide about how secure your DUCC should be. We lock DUCC's broker by > default to protect it from a malicious user code. If this is not your > concern you can use DUCC's broker if you want. Keep in mind that if you use > DUCC's broker for services its load will increase and if this is high > enough it may effect how DUCC responds. Another benefit of having a > secondary broker is that you can keep your services up while DUCC is being > shutdown for update or maintenance. This is especially important if your > services take hours to initialize. > > Jerry > > On Tue, Jul 18, 2017 at 4:07 PM, Osborne, John David (Campus) < > [email protected]> wrote: > > > Thanks Jerry! > > > > I can run it using the insecure option (just tried it - works great!) > > - the broker port is closed to all but localhost as another > > application feeds requests to DUCC. From a non-security operational > > point of few though I am violating your recommendation to use a > > secondary broker... I hope this won't get me into too much trouble... > > > > From the documentation, I had no idea there was a need for a secondary > > broker to run services. If I wanted to do this, is this something that > > can be set up in the current ducc distribution or would it be better > > to go ahead and run my legacy services independently? To do this, I'm > > presuming I would instantiate a secondary broker (presumably still > > 61616) but submit the deployment descriptor to the DUCC broker (61617) > > but with the inputQueue brokerURL pointing to the secondary (61616) > broker? > > > > I was hoping to make a clean migration to DUCC - maybe it would be > > easier to get rid of services altogether and just submit jobs? Is this > > the current trend or are a lot of folks running secondary brokers and > services? > > > > -John > > > > > > > > -----Original Message----- > > From: [email protected] [mailto:[email protected]] On Behalf Of Jaroslaw > > Cwiklik > > Sent: Tuesday, July 18, 2017 2:19 PM > > To: [email protected] > > Subject: Re: DUCC Security Model for Service Deployment > > > > Hi, as Lou stated its recommended to use a secondary broker for > > services to keep this separate from a DUCC broker. The DUCC broker is > > protected by default as it is used for internal communication. > > > > If you want to run without broker credentials change this setting in > > default.ducc.properties > > > > ducc.broker.configuration = conf/activemq-ducc-unsecure.xml > > > > Jerry > > > > > > > > On Tue, Jul 18, 2017 at 3:05 PM, Osborne, John David (Campus) < > > [email protected]> wrote: > > > > > Thanks Lou, > > > > > > I'm still not totally following. > > > > > > In UIMA-AS, I could deploy services (as the administrator) using: > > > deployAsyncService.sh $SOME_DEPLOYMENT_DESCRIPTOR $BROKER_URL > > > > > > I thought the equivalent in DUCC was (for example): ducc_services > > > --register --process_descriptor $SOME_DEPLOYMENT_DESCRIPTOR > > > --classpath $CLASSPATH --autostart true > > > > > > In the first case, I don't recall worrying about passwords - I > > > presume deployAsyncService was privileged and knew any broker > > > credentials. Are you saying the command line utility ducc_services > > > is unaware of ducc-broker-credentionals.properties and there is no > > > other way for the ducc user to pass these credentionals to the > > > broker when registering/auto-start services? > > > > > > Sorry for the confusion, new to DUCC. I can see the > > > $DUCC_HOME/resources.private/ducc-broker-credentials.properties - I > > > just don't know how to use them when deploying (same thing as > > > registering and > > > starting?) a service. > > > > > > -John > > > > > > > > > > > > -----Original Message----- > > > From: Lou DeGenaro [mailto:[email protected]] > > > Sent: Tuesday, July 18, 2017 1:34 PM > > > To: [email protected] > > > Subject: Re: DUCC Security Model for Service Deployment > > > > > > The DUCC broker is for DUCC. User services should employ a separate > > > user broker, not the DUCC broker. > > > > > > That said, with proper permissions you can discover the DUCC broker > > > user and pw in > > > $DUCC_HOME/resources.private/ducc-broker-credentials.properties > > > > > > Lou. > > > > > > On Tue, Jul 18, 2017 at 2:14 PM, Osborne, John David (Campus) < > > > [email protected]> wrote: > > > > > > > Yes - the idea (based on a previous UIMA-AS workflow) was to > > > > attached services to the broker and have other application/s call > > > > them > > as needed. > > > > > > > > > > > > I thought this was still done with DUCC? > > > > > > > > > > > > I see what I *think* is the active security configuration in > > > > activemq.xml > > > > below: > > > > > > > > > > > > <simpleAuthenticationPlugin anonymousAccessAllowed="false"> > > > > <users> > > > > <authenticationUser username="${ducc.broker.admin. > > > username}" > > > > password="${ducc.broker.admin.password}" > > > > groups="ducc-admin"/> > > > > </users> > > > > </simpleAuthenticationPlugin> > > > > > > > > > > > > I'm registering services and running ducc as user ducc. > > > > > > > > > > > > -John > > > > > > > > > > > > ________________________________ > > > > From: Lou DeGenaro <[email protected]> > > > > Sent: Tuesday, July 18, 2017 12:19:21 PM > > > > To: [email protected] > > > > Subject: Re: DUCC Security Model for Service Deployment > > > > > > > > DUCC has its own password protected AMQ broker used for daemon > > > > communications. Are your services trying to use DUCC's broker? > > > > > > > > Lou. > > > > > > > > On Tue, Jul 18, 2017 at 1:02 PM, Osborne, John David (Campus) < > > > > [email protected]> wrote: > > > > > > > > > I am having deploying services with DUCC using a single system, > > > > > single user setup. I can run test jobs, so I *think* my system > > > > > is set up > > > > correctly > > > > > but I can't register and then start services due to > > > > > "SecurityException on Connect". > > > > > > > > > > > > > > > I don't understand how when registering services the > > > > > credentionals are passed to DUCC, I know it uses activemq > > > > > credentionals, but there is no parameter to pass them on the > command line. > > > > > > > > > > > > > > > I am registering the service like this: > > > > > > > > > > /web/servers/apache-uima-ducc-2.2.0/bin/ducc_services --register > > > > > --process_descriptor_DD ../resources/desc/deployment/ > > > > > ImportDocumentsDeploymentDescriptorSimple.xml --description > > > > > "Pull Documents" --classpath $CLASSPATH --autostart true > > > > > > > > > > > > > > > I have not messed with any of the configuration files > > > > > (credentional.properties, etc...) under apache-activemq. > > > > > > > > > > > > > > > My error (actually a warning) is below: > > > > > > > > > > > > > > > Any help appreciated, > > > > > > > > > > > > > > > -John > > > > > > > > > > > > > > > INFO: Controller: MedicsClobsConsumer Trying to Start Listener > > > > > on > > > > > Endpoint: queue://MRN_Document_Pull_Queue Selector: Command=2001 > > > Broker: > > > > > tcp://localhost:61617 > > > > > >>> Service Container Deployed Successfully > > > > > DuccAbstractProcessContainer.deploy() <<<<<<<< User Container > > > > > deployed .... Deployed Processing Container - Initialization > > > > > Successful - Thread 1 > > > > > 18 Jul 2017 11:40:22,858 INFO AgentSession - T[1] > > > > > notifyAgentWithStatus ... Job Process State Changed - PID:5803. > > > > > Process State: Running. JMX > > > > > Url:service:jmx:rmi:///jndi/rmi://uimaprapp1.hs.uab.edu:2105/jmx > > > > > rm i Dispatched State Update Event to Agent with > > > > > IP:10.23.142.165 Jul 18, > > > > > 2017 11:40:23 AM org.apache.uima.adapter.jms.activemq. > > > > > UimaDefaultMessageListenerContainer onException > > > > > WARNING: Service: MedicsClobsConsumer Runtime Exception Jul 18, > > > > > 2017 > > > > > 11:40:23 AM org.apache.uima.adapter.jms.activemq. > > > > > UimaDefaultMessageListenerContainer onException > > > > > WARNING: Jms Listener Failed. Endpoint: MRN_Document_Pull_Queue > > > > > Managed > > > > > By: tcp://localhost:61617 Reason: org.apache.activemq. > > > > ConnectionFailedException: > > > > > The JMS connection has failed: Force close due to > > > > > SecurityException on connect Jul 18, 2017 11:40:23 AM > > > > > org.apache.uima.adapter.jms.activemq. > > > > > UimaDefaultMessageListenerContainer handleListenerSetupFailure > > > > > WARNING: Uima AS Service:MedicsClobsConsumer Listener Unable To > > > > > Connect > > > > To > > > > > Broker: tcp://localhost:61617 Retrying Until Successful ... > > > > > > > > > > > > > > > > > > > > > > > > >
