Jerry, my services would span across multiple jobs so a service model is preferable for me.
With regards to a secondary broker - I'm thinking to use a UIMA-AS installation on the same machine rather than download the full Apache AMQ runtime standalone. I'm worried about potential compatibility issues between DUCC broker and downloaded broker (unfounded concern?) and I am unfamiliar with AMQ in isolation. It should also allow for comparison between AS and DUCC. Is this a reasonable approach? Thanks for all your help so far, -John -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Jaroslaw Cwiklik Sent: Tuesday, July 18, 2017 3:40 PM To: [email protected] Subject: Re: DUCC Security Model for Service Deployment John, services are typically long running and independent of jobs. They provide a specific function and may take a long time to initialize. They are meant to be reusable across multiple jobs. If your analytics are fast initializing and not meant to be shared you can just use jobs. We should probably modify the documentation to talk about recommending a secondary broker for services. My recommendation is to download and install the full AMQ runtime for your services. DUCC only bundles essential parts of AMQ distro. You decide about how secure your DUCC should be. We lock DUCC's broker by default to protect it from a malicious user code. If this is not your concern you can use DUCC's broker if you want. Keep in mind that if you use DUCC's broker for services its load will increase and if this is high enough it may effect how DUCC responds. Another benefit of having a secondary broker is that you can keep your services up while DUCC is being shutdown for update or maintenance. This is especially important if your services take hours to initialize. Jerry On Tue, Jul 18, 2017 at 4:07 PM, Osborne, John David (Campus) < [email protected]> wrote: > Thanks Jerry! > > I can run it using the insecure option (just tried it - works great!) > - the broker port is closed to all but localhost as another > application feeds requests to DUCC. From a non-security operational > point of few though I am violating your recommendation to use a > secondary broker... I hope this won't get me into too much trouble... > > From the documentation, I had no idea there was a need for a secondary > broker to run services. If I wanted to do this, is this something that > can be set up in the current ducc distribution or would it be better > to go ahead and run my legacy services independently? To do this, I'm > presuming I would instantiate a secondary broker (presumably still > 61616) but submit the deployment descriptor to the DUCC broker (61617) > but with the inputQueue brokerURL pointing to the secondary (61616) broker? > > I was hoping to make a clean migration to DUCC - maybe it would be > easier to get rid of services altogether and just submit jobs? Is this > the current trend or are a lot of folks running secondary brokers and > services? > > -John > > > > -----Original Message----- > From: [email protected] [mailto:[email protected]] On Behalf Of Jaroslaw > Cwiklik > Sent: Tuesday, July 18, 2017 2:19 PM > To: [email protected] > Subject: Re: DUCC Security Model for Service Deployment > > Hi, as Lou stated its recommended to use a secondary broker for > services to keep this separate from a DUCC broker. The DUCC broker is > protected by default as it is used for internal communication. > > If you want to run without broker credentials change this setting in > default.ducc.properties > > ducc.broker.configuration = conf/activemq-ducc-unsecure.xml > > Jerry > > > > On Tue, Jul 18, 2017 at 3:05 PM, Osborne, John David (Campus) < > [email protected]> wrote: > > > Thanks Lou, > > > > I'm still not totally following. > > > > In UIMA-AS, I could deploy services (as the administrator) using: > > deployAsyncService.sh $SOME_DEPLOYMENT_DESCRIPTOR $BROKER_URL > > > > I thought the equivalent in DUCC was (for example): ducc_services > > --register --process_descriptor $SOME_DEPLOYMENT_DESCRIPTOR > > --classpath $CLASSPATH --autostart true > > > > In the first case, I don't recall worrying about passwords - I > > presume deployAsyncService was privileged and knew any broker > > credentials. Are you saying the command line utility ducc_services > > is unaware of ducc-broker-credentionals.properties and there is no > > other way for the ducc user to pass these credentionals to the > > broker when registering/auto-start services? > > > > Sorry for the confusion, new to DUCC. I can see the > > $DUCC_HOME/resources.private/ducc-broker-credentials.properties - I > > just don't know how to use them when deploying (same thing as > > registering and > > starting?) a service. > > > > -John > > > > > > > > -----Original Message----- > > From: Lou DeGenaro [mailto:[email protected]] > > Sent: Tuesday, July 18, 2017 1:34 PM > > To: [email protected] > > Subject: Re: DUCC Security Model for Service Deployment > > > > The DUCC broker is for DUCC. User services should employ a separate > > user broker, not the DUCC broker. > > > > That said, with proper permissions you can discover the DUCC broker > > user and pw in > > $DUCC_HOME/resources.private/ducc-broker-credentials.properties > > > > Lou. > > > > On Tue, Jul 18, 2017 at 2:14 PM, Osborne, John David (Campus) < > > [email protected]> wrote: > > > > > Yes - the idea (based on a previous UIMA-AS workflow) was to > > > attached services to the broker and have other application/s call > > > them > as needed. > > > > > > > > > I thought this was still done with DUCC? > > > > > > > > > I see what I *think* is the active security configuration in > > > activemq.xml > > > below: > > > > > > > > > <simpleAuthenticationPlugin anonymousAccessAllowed="false"> > > > <users> > > > <authenticationUser username="${ducc.broker.admin. > > username}" > > > password="${ducc.broker.admin.password}" > > > groups="ducc-admin"/> > > > </users> > > > </simpleAuthenticationPlugin> > > > > > > > > > I'm registering services and running ducc as user ducc. > > > > > > > > > -John > > > > > > > > > ________________________________ > > > From: Lou DeGenaro <[email protected]> > > > Sent: Tuesday, July 18, 2017 12:19:21 PM > > > To: [email protected] > > > Subject: Re: DUCC Security Model for Service Deployment > > > > > > DUCC has its own password protected AMQ broker used for daemon > > > communications. Are your services trying to use DUCC's broker? > > > > > > Lou. > > > > > > On Tue, Jul 18, 2017 at 1:02 PM, Osborne, John David (Campus) < > > > [email protected]> wrote: > > > > > > > I am having deploying services with DUCC using a single system, > > > > single user setup. I can run test jobs, so I *think* my system > > > > is set up > > > correctly > > > > but I can't register and then start services due to > > > > "SecurityException on Connect". > > > > > > > > > > > > I don't understand how when registering services the > > > > credentionals are passed to DUCC, I know it uses activemq > > > > credentionals, but there is no parameter to pass them on the command > > > > line. > > > > > > > > > > > > I am registering the service like this: > > > > > > > > /web/servers/apache-uima-ducc-2.2.0/bin/ducc_services --register > > > > --process_descriptor_DD ../resources/desc/deployment/ > > > > ImportDocumentsDeploymentDescriptorSimple.xml --description > > > > "Pull Documents" --classpath $CLASSPATH --autostart true > > > > > > > > > > > > I have not messed with any of the configuration files > > > > (credentional.properties, etc...) under apache-activemq. > > > > > > > > > > > > My error (actually a warning) is below: > > > > > > > > > > > > Any help appreciated, > > > > > > > > > > > > -John > > > > > > > > > > > > INFO: Controller: MedicsClobsConsumer Trying to Start Listener > > > > on > > > > Endpoint: queue://MRN_Document_Pull_Queue Selector: Command=2001 > > Broker: > > > > tcp://localhost:61617 > > > > >>> Service Container Deployed Successfully > > > > DuccAbstractProcessContainer.deploy() <<<<<<<< User Container > > > > deployed .... Deployed Processing Container - Initialization > > > > Successful - Thread 1 > > > > 18 Jul 2017 11:40:22,858 INFO AgentSession - T[1] > > > > notifyAgentWithStatus ... Job Process State Changed - PID:5803. > > > > Process State: Running. JMX > > > > Url:service:jmx:rmi:///jndi/rmi://uimaprapp1.hs.uab.edu:2105/jmx > > > > rm i Dispatched State Update Event to Agent with > > > > IP:10.23.142.165 Jul 18, > > > > 2017 11:40:23 AM org.apache.uima.adapter.jms.activemq. > > > > UimaDefaultMessageListenerContainer onException > > > > WARNING: Service: MedicsClobsConsumer Runtime Exception Jul 18, > > > > 2017 > > > > 11:40:23 AM org.apache.uima.adapter.jms.activemq. > > > > UimaDefaultMessageListenerContainer onException > > > > WARNING: Jms Listener Failed. Endpoint: MRN_Document_Pull_Queue > > > > Managed > > > > By: tcp://localhost:61617 Reason: org.apache.activemq. > > > ConnectionFailedException: > > > > The JMS connection has failed: Force close due to > > > > SecurityException on connect Jul 18, 2017 11:40:23 AM > > > > org.apache.uima.adapter.jms.activemq. > > > > UimaDefaultMessageListenerContainer handleListenerSetupFailure > > > > WARNING: Uima AS Service:MedicsClobsConsumer Listener Unable To > > > > Connect > > > To > > > > Broker: tcp://localhost:61617 Retrying Until Successful ... > > > > > > > > > > > > > > > > > >
