[jira] [Created] (APEXCORE-796) Docker based deployment

[Thomas Weise (JIRA)]

Thomas Weise created APEXCORE-796:
-

 Summary: Docker based deployment
 Key: APEXCORE-796
 URL: https://issues.apache.org/jira/browse/APEXCORE-796
 Project: Apache Apex Core
  Issue Type: New Feature
Reporter: Thomas Weise


Apex should support deployment using Docker as alternative to application 
packages. Docker images provide a simple and standard way to package 
dependencies and solve isolation from the host environment. This will be 
particularly helpful when applications depend on native, non-JVM packages like 
Python and R, that otherwise need to be installed separately. Docker support 
will also be a step towards supporting other cluster managers like Kubernetes, 
Mesos and Swarm.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (APEXCORE-781) Autometric values of an operator is showing wrongly in App master

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249694#comment-16249694
 ] 

ASF GitHub Bot commented on APEXCORE-781:
-

vrozov closed pull request #578: APEXCORE-781 Update the autometrics only if 
the number of partitions matches with the size of metric collection
URL: https://github.com/apache/apex-core/pull/578
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java 
b/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java
index 07641d23e2..1566d2dd66 100644
--- a/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java
+++ b/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java
@@ -258,6 +258,7 @@
   private final Map> latestLogicalMetrics = 
Maps.newHashMap();
+  private final Map>> 
latestLogicalCompletedMetrics = Maps.newHashMap();
 
   //logical operator name to latest counters. exists for backward 
compatibility.
   private final Map latestLogicalCounters = Maps.newHashMap();
@@ -934,6 +935,7 @@ private void aggregateMetrics(long windowId, Map endWin
   metricPool.add(physicalMetrics);
 }
   }
+
   if (metricPool.isEmpty()) {
 //nothing to aggregate
 continue;
@@ -960,6 +962,9 @@ public boolean add(Pair> 
longMapPair)
 }
 LOG.debug("Adding to logical metrics for {}", operatorMeta.getName());
 windowMetrics.add(new Pair<>(windowId, lm));
+if (metricPool.size() == physicalOperators.size()) {
+  latestLogicalCompletedMetrics.put(operatorMeta.getName(), new 
Pair<>(windowId, lm));
+}
 Map oldValue = 
latestLogicalMetrics.put(operatorMeta.getName(), lm);
 if (oldValue == null) {
   try {
@@ -2495,11 +2500,16 @@ public OperatorInfo getOperatorInfo(int operatorId)
 
   public LogicalOperatorInfo getLogicalOperatorInfo(String operatorName)
   {
+return getLogicalOperatorInfo(operatorName, true);
+  }
+
+  public LogicalOperatorInfo getLogicalOperatorInfo(String operatorName, 
boolean isPartialAggregates)
+  {
 OperatorMeta operatorMeta = getLogicalPlan().getOperatorMeta(operatorName);
 if (operatorMeta == null) {
   return null;
 }
-return fillLogicalOperatorInfo(operatorMeta);
+return fillLogicalOperatorInfo(operatorMeta, isPartialAggregates);
   }
 
   public ModuleMeta getModuleMeta(String moduleName)
@@ -2526,7 +2536,7 @@ private ModuleMeta getModuleMeta(String moduleName, 
LogicalPlan dag)
 List infoList = new ArrayList<>();
 Collection allOperators = getLogicalPlan().getAllOperators();
 for (OperatorMeta operatorMeta : allOperators) {
-  infoList.add(fillLogicalOperatorInfo(operatorMeta));
+  infoList.add(getLogicalOperatorInfo(operatorMeta.getName()));
 }
 return infoList;
   }
@@ -2608,7 +2618,7 @@ private OperatorInfo fillPhysicalOperatorInfo(PTOperator 
operator)
 return oi;
   }
 
-  private LogicalOperatorInfo fillLogicalOperatorInfo(OperatorMeta operator)
+  private LogicalOperatorInfo fillLogicalOperatorInfo(OperatorMeta operator, 
boolean isPartialAggregates)
   {
 LogicalOperatorInfo loi = new LogicalOperatorInfo();
 loi.name = operator.getName();
@@ -2673,7 +2683,12 @@ private LogicalOperatorInfo 
fillLogicalOperatorInfo(OperatorMeta operator)
 if (physicalOperators.size() > 0 && checkpointTimeAggregate.getAvg() != 
null) {
   loi.checkpointTimeMA = checkpointTimeAggregate.getAvg().longValue();
   loi.counters = latestLogicalCounters.get(operator.getName());
-  loi.autoMetrics = latestLogicalMetrics.get(operator.getName());
+  if (isPartialAggregates) {
+loi.autoMetrics = latestLogicalMetrics.get(operator.getName());
+  } else {
+loi.autoMetrics = 
latestLogicalCompletedMetrics.get(operator.getName());
+  }
+
 }
 
 return loi;
diff --git 
a/engine/src/main/java/com/datatorrent/stram/webapp/StramWebServices.java 
b/engine/src/main/java/com/datatorrent/stram/webapp/StramWebServices.java
index 995127c1b6..5e0eea3a84 100644
--- a/engine/src/main/java/com/datatorrent/stram/webapp/StramWebServices.java
+++ b/engine/src/main/java/com/datatorrent/stram/webapp/StramWebServices.java
@@ -574,6 +574,21 @@ public JSONObject getLogicalOperators() throws Exception
   }
 
   @GET
+  

[ANNOUNCE] Apache Apex Malhar 3.8.0 released

[Ananth G]

Dear Community,

The Apache Apex community is pleased to announce release 3.8.0 of the Apex 
Malhar library.

Apache Apex is an enterprise grade big data-in-motion platform that unifies 
stream and batch processing. Apex was built for scalability and low-latency 
processing, high availability and operability. The Apex engine is supplemented 
by Malhar, the library of pre-built operators, including connectors that 
integrate with many existing technologies as sources and destinations, like 
message buses, databases, files or social media feeds.
Along with bug fixes, this release brings in improved support for Flume 
operator, bloom filter support for bucketing, better failure handling in kafka 
input operator and multi Hbase table output operator among others.

New features include support for Kudu, support for sort accumulation in 
Windowed Operator besides enhancing some example applications.

Changes: https://github.com/apache/apex-malhar/blob/v3.8.0/CHANGELOG.md 


The source release can be found at: 
http://www.apache.org/dyn/closer.lua/apex/apache-apex-malhar-3.8.0/apache-apex-malhar-3.8.0-source-release.tar.gz
 


or visit: http://apex.apache.org/downloads.html 


We welcome your help and feedback. For more information on the project and how 
to get involved, visit our website at: http://apex.apache.org/

Regards,
The Apache Apex community