[jira] [Commented] (AIRAVATA-2054) Simple version of docker based airavata run
[ https://issues.apache.org/jira/browse/AIRAVATA-2054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15446209#comment-15446209 ] ASF subversion and git services commented on AIRAVATA-2054: --- Commit 91afea32dee2770e05447716ae29f3ce7655e36d in airavata's branch refs/heads/lahiru/airavata-docker from Lahiru Ginnaliya Gamathige [ https://git-wip-us.apache.org/repos/asf?p=airavata.git;h=91afea3 ] [AIRAVATA-2054][WIP] create docker images for airavata deployment components 1. Introduce Docker images for each deployment component of airavata. 2. Deployed those in docker hub repository (scigap), try: docker search scigap 3. Use exhibitor docker images intead of zookeeper which is a much better compare to using vanilla zookeeper. http://techblog.netflix.com/2012/04/introducing-exhibitor-supervisor-system.html 4. IMHO we should never use docker images from public repository, Everything we should create our own docker images from public images and test with those and move to production. 5. Added a simple script(airavata/build.sh) to build airavata docker components. ./build.sh [component-name] - This will build a docker image for given component. This is a temporary script we can use until AIRAVATA-2056 which integrates docker push with some CI tool like jenkins. > Simple version of docker based airavata run > --- > > Key: AIRAVATA-2054 > URL: https://issues.apache.org/jira/browse/AIRAVATA-2054 > Project: Airavata > Issue Type: Sub-task >Reporter: Lahiru Gunathilake >Assignee: Lahiru Gunathilake > > When airavata-server.properties configured properly for known setup build > airavata docker image and other dependent docker images (exhibitor, rabbitmq, > mysql) and deploy the complete setup. > This is a simple version where developer has to manually checkout the git > branch and configure the properties and build docker images and deploy the > changes in a VM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRAVATA-2056) Jenkins bulid script for airavata push
[ https://issues.apache.org/jira/browse/AIRAVATA-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15446210#comment-15446210 ] ASF subversion and git services commented on AIRAVATA-2056: --- Commit 91afea32dee2770e05447716ae29f3ce7655e36d in airavata's branch refs/heads/lahiru/airavata-docker from Lahiru Ginnaliya Gamathige [ https://git-wip-us.apache.org/repos/asf?p=airavata.git;h=91afea3 ] [AIRAVATA-2054][WIP] create docker images for airavata deployment components 1. Introduce Docker images for each deployment component of airavata. 2. Deployed those in docker hub repository (scigap), try: docker search scigap 3. Use exhibitor docker images intead of zookeeper which is a much better compare to using vanilla zookeeper. http://techblog.netflix.com/2012/04/introducing-exhibitor-supervisor-system.html 4. IMHO we should never use docker images from public repository, Everything we should create our own docker images from public images and test with those and move to production. 5. Added a simple script(airavata/build.sh) to build airavata docker components. ./build.sh [component-name] - This will build a docker image for given component. This is a temporary script we can use until AIRAVATA-2056 which integrates docker push with some CI tool like jenkins. > Jenkins bulid script for airavata push > -- > > Key: AIRAVATA-2056 > URL: https://issues.apache.org/jira/browse/AIRAVATA-2056 > Project: Airavata > Issue Type: Sub-task >Reporter: Lahiru Gunathilake >Assignee: Lahiru Gunathilake > > When someone gives the branch and other parameters (airavata mode : all, > orchestrator, gfac, api-server, environment-name : nightly, staging, > production) compile that branch and create docker image and push those to > docker registry. > So if we have already setup the environment with systemd units, we just have > to restart the systemd unit after running this jenkins script, it will pull > the latest changes and depoy all the changes for your branch. > We don't need jenkins for other depenent components like exhibitor, rabbitmq. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (AIRAVATA-2068) Pump logs to kafka topic
[
https://issues.apache.org/jira/browse/AIRAVATA-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lahiru Gunathilake resolved AIRAVATA-2068.
--
Resolution: Fixed
Kafka logging is turned off by default, following are the kafka properties need
to configure.
# Kafka Logging related configuration
isRunningOnAws= false
kafka.broker.list= 54.8.14.132:9092
kafka.topic.prefix= local
enable.kafka.logging= true
If you setup a kafka cluster and listen to the topic airavata is pushing (Its
created dynamically based on airavata role and kafka.topic.prefix configured)
you'll see logs are published like this.
{"serverId":{"serverId":"192.168.0.14","hostName":"192.168.0.14","version":"airavata-0.16-128-g415e9b7","roles":["all"]},"message":"Reading
configuration from:
/Users/lahiru/iu/airavata/modules/distribution/target/apache-airavata-server-0.17-SNAPSHOT/bin/zoo.cfg","timestamp":"2016-08-29T14:13:35.185Z","level":"INFO","loggerName":"org.apache.zookeeper.server.quorum.QuorumPeerConfig","mdc":{},"threadName":"main"}
{"serverId":{"serverId":"192.168.0.14","hostName":"192.168.0.14","version":"airavata-0.16-128-g415e9b7","roles":["all"]},"message":"Airavata
server instance
starting...","timestamp":"2016-08-29T14:13:35.552Z","level":"INFO","loggerName":"org.apache.airavata.server.ServerMain","mdc":{},"threadName":"main"}
{"serverId":{"serverId":"192.168.0.14","hostName":"192.168.0.14","version":"airavata-0.16-128-g415e9b7","roles":["all"]},"message":"Server
environment:zookeeper.version\u003d3.4.0-1202560, built on 11/16/2011 07:18
GMT","timestamp":"2016-08-29T14:13:35.592Z","level":"INFO","loggerName":"org.apache.zookeeper.server.ZooKeeperServer","mdc":{},"threadName":"Thread-1"}
{"serverId":{"serverId":"192.168.0.14","hostName":"192.168.0.14","version":"airavata-0.16-128-g415e9b7","roles":["all"]},"message":"Server
environment:java.version\u003d1.8.0_45","timestamp":"2016-08-29T14:13:35.594Z","level":"INFO","loggerName":"org.apache.zookeeper.server.ZooKeeperServer","mdc":{},"threadName":"Thread-1"}
{"serverId":{"serverId":"192.168.0.14","hostName":"192.168.0.14","version":"airavata-0.16-128-g415e9b7","roles":["all"]},"message":"Server
environment:java.home\u003d/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre","timestamp":"2016-08-29T14:13:35.596Z","level":"INFO","loggerName":"org.apache.zookeeper.server.ZooKeeperServer","mdc":{},"threadName":"Thread-1"}
{"serverId":{"serverId":"192.168.0.14","hostName":"192.168.0.14","version":"airavata-0.16-128-g415e9b7","roles":["all"]},"message":"Server
environment:java.library.path\u003d/Users/lahiru/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.","timestamp":"2016-08-29T14:13:35.602Z","level":"INFO","loggerName":"org.apache.zookeeper.server.ZooKeeperServer","mdc":{},"threadName":"Thread-1"}
{"serverId":{"serverId":"192.168.0.14","hostName":"192.168.0.14","version":"airavata-0.16-128-g415e9b7","roles":["all"]},"message":"Server
environment:java.compiler\u003d\u003cNA\u003e","timestamp":"2016-08-29T14:13:35.603Z","level":"INFO","loggerName":"org.apache.zookeeper.server.ZooKeeperServer","mdc":{},"threadName":"Thread-1"}
> Pump logs to kafka topic
> -
>
> Key: AIRAVATA-2068
> URL: https://issues.apache.org/jira/browse/AIRAVATA-2068
> Project: Airavata
> Issue Type: Sub-task
>Reporter: Lahiru Gunathilake
>Assignee: Lahiru Gunathilake
>
> Make kafka logging configuration (turn it off by default)
> Configure kafka configurations
> Set the appender to org.apache.airavata
> Use logback instead of log4j
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (AIRAVATA-2072) Create MDC context to store airavata context
Lahiru Gunathilake created AIRAVATA-2072: Summary: Create MDC context to store airavata context Key: AIRAVATA-2072 URL: https://issues.apache.org/jira/browse/AIRAVATA-2072 Project: Airavata Issue Type: Sub-task Reporter: Lahiru Gunathilake Create MDC so that airavata context can be injected, ex: experimentId So that we can query the logs specific to a given experiment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AIRAVATA-2071) View airavata logs in web GUI
Lahiru Gunathilake created AIRAVATA-2071: Summary: View airavata logs in web GUI Key: AIRAVATA-2071 URL: https://issues.apache.org/jira/browse/AIRAVATA-2071 Project: Airavata Issue Type: Sub-task Reporter: Lahiru Gunathilake Find a good way of viewing Airavata logs and create indexes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AIRAVATA-2070) Create elastic search service to keep the logs
Lahiru Gunathilake created AIRAVATA-2070: Summary: Create elastic search service to keep the logs Key: AIRAVATA-2070 URL: https://issues.apache.org/jira/browse/AIRAVATA-2070 Project: Airavata Issue Type: Sub-task Reporter: Lahiru Gunathilake Use Qbox of AWS elastic search to store the logs, test the full path of logs flow where airavata -> kafka -> logstash -> ES. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AIRAVATA-2069) Create logstash to pull the kafka logging
Lahiru Gunathilake created AIRAVATA-2069: Summary: Create logstash to pull the kafka logging Key: AIRAVATA-2069 URL: https://issues.apache.org/jira/browse/AIRAVATA-2069 Project: Airavata Issue Type: Sub-task Reporter: Lahiru Gunathilake Create a logstash configuration and logstash docker image to pull logs from kafka topic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
