apex-malhar git commit: APEXMALHAR-2433 #resolve #comment Add readme for Windowing Benchmark closes #573

2017-03-20 Thread thw 
Repository: apex-malhar
Updated Branches:
  refs/heads/master d45e8369c -> 2b3c2aa88


APEXMALHAR-2433 #resolve #comment Add readme for Windowing Benchmark
closes #573


Project: http://git-wip-us.apache.org/repos/asf/apex-malhar/repo
Commit: http://git-wip-us.apache.org/repos/asf/apex-malhar/commit/2b3c2aa8
Tree: http://git-wip-us.apache.org/repos/asf/apex-malhar/tree/2b3c2aa8
Diff: http://git-wip-us.apache.org/repos/asf/apex-malhar/diff/2b3c2aa8

Branch: refs/heads/master
Commit: 2b3c2aa8884fa946dc25c91e17577e3ac6e7964b
Parents: d45e836
Author: brightchen 
Authored: Mon Mar 6 13:33:51 2017 -0800
Committer: Thomas Weise 
Committed: Mon Mar 20 17:16:15 2017 -0700

--
 .../com/datatorrent/benchmark/window/README.md  | 32 
 1 file changed, 32 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/apex-malhar/blob/2b3c2aa8/benchmark/src/main/java/com/datatorrent/benchmark/window/README.md
--
diff --git a/benchmark/src/main/java/com/datatorrent/benchmark/window/README.md 
b/benchmark/src/main/java/com/datatorrent/benchmark/window/README.md
new file mode 100644
index 000..604b43b
--- /dev/null
+++ b/benchmark/src/main/java/com/datatorrent/benchmark/window/README.md
@@ -0,0 +1,32 @@
+Windowed Operator Benchmark Readme
+
+Introduction
+The Windowed operator benchmark are applications to test the performance of 
windowed operator. 
+This benchmark applications only available for Malhar 3.7.0 or later versions. 
+The source code is located inside the malhar/benchmark. There are two 
applications, 
+one is benchmark for WindowedOperator, another is for KeyedWindowedOperator. 
+
+Steps of compile
+- Get source code of malhar
+- Compile the malhar with all module option "-Pall-modules": "mvn clean 
install -Pall-modules -DskipTests"
+
+Run the benchmark and check the result in cluster
+- Start apex cli
+- Launch application "WindowedOperatorBenchmarkApp" or 
"KeyedWindowedOperatorBenchmarkApp". Get application id, for example 
"application_1487803614053_10401"
+- Use command "connect" to connect to the application
+- Use command "list-operators" to list operators of the application
+- Find the operator with name "output". Get the value of "host" field. for 
example "node1:8041"
+- Go to the resource manager and click "Nodes" link in the left panel
+- Find the node by "Node Address" which run the operator "output", click the 
link of "Node HTTP Address" column. for example "node1:8042"
+- Expand the "Tools" on the left panel and click "Local logs"
+- Click the "container/" link
+- Select the application folder. The folder name is same as the application id
+- Click the container and then click the log file 
+- Open the log. you will find the benchmark which log in the log file for 
example: 
+  "total: count: 7356000; time: 278453; average: 26417; period: count: 40; 
time: 10005; average: 39980"
+  Each log of benchmark includes “total” section and “period” section. 
+  Total section counts all time from application started to log time. 
+  "period" counts the time from previous log time to current log time. 
+- The period average can help to determine if the rate is stable or not. The 
total average suppose to trends to the period average if application is stable
+  The valid rate should be the stable rate
+- Suggest to run this application several times to eliminate any temporary 
interference by environment.

[2/2] apex-malhar git commit: Merge commit 'refs/pull/545/head' of github.com:apache/apex-malhar

2017-03-20 Thread chinmay 
Merge commit 'refs/pull/545/head' of github.com:apache/apex-malhar


Project: http://git-wip-us.apache.org/repos/asf/apex-malhar/repo
Commit: http://git-wip-us.apache.org/repos/asf/apex-malhar/commit/d45e8369
Tree: http://git-wip-us.apache.org/repos/asf/apex-malhar/tree/d45e8369
Diff: http://git-wip-us.apache.org/repos/asf/apex-malhar/diff/d45e8369

Branch: refs/heads/master
Commit: d45e8369cda6449ed5da6793ea62a70972c2c71c
Parents: 89b2937 f29c7d4
Author: Chinmay Kolhatkar 
Authored: Mon Mar 20 13:46:34 2017 +0530
Committer: Chinmay Kolhatkar 
Committed: Mon Mar 20 13:46:34 2017 +0530

--
 .../malhar/contrib/parser/CommonLogParser.java  |  84 +
 .../malhar/contrib/parser/log/CommonLog.java| 120 ++
 .../contrib/parser/CommonLogParserTest.java | 124 +++
 3 files changed, 328 insertions(+)
--

apex-core git commit: APEXCORE-580 APEXCORE-581 Renamed UserDefinedControlTuple to ControlTuple, moved to different package

2017-03-20 Thread pramod 
Repository: apex-core
Updated Branches:
  refs/heads/master 576047e41 -> d6f17f23a


APEXCORE-580 APEXCORE-581 Renamed UserDefinedControlTuple to ControlTuple, 
moved to different package


Project: http://git-wip-us.apache.org/repos/asf/apex-core/repo
Commit: http://git-wip-us.apache.org/repos/asf/apex-core/commit/d6f17f23
Tree: http://git-wip-us.apache.org/repos/asf/apex-core/tree/d6f17f23
Diff: http://git-wip-us.apache.org/repos/asf/apex-core/diff/d6f17f23

Branch: refs/heads/master
Commit: d6f17f23a9c11e49a72dc87f76908c71ec235b11
Parents: 576047e
Author: bhupeshchawda 
Authored: Mon Mar 6 18:14:21 2017 +0530
Committer: bhupeshchawda 
Committed: Sun Mar 19 18:54:50 2017 +0530

--
 .../api/ControlTupleEnabledSink.java|  6 +--
 .../apex/api/ControlAwareDefaultInputPort.java  |  5 ++-
 .../apex/api/ControlAwareDefaultOutputPort.java |  9 ++--
 .../apex/api/UserDefinedControlTuple.java   | 46 
 .../apache/apex/api/operator/ControlTuple.java  | 46 
 .../datatorrent/stram/engine/GenericNode.java   | 10 ++---
 .../com/datatorrent/stram/engine/OiONode.java   | 10 ++---
 .../stram/engine/WindowGenerator.java   |  4 +-
 .../stram/stream/BufferServerPublisher.java |  4 +-
 .../stram/stream/BufferServerSubscriber.java|  4 +-
 .../datatorrent/stram/stream/FastPublisher.java |  4 +-
 .../datatorrent/stram/stream/InlineStream.java  |  4 +-
 .../com/datatorrent/stram/stream/MuxStream.java |  4 +-
 .../com/datatorrent/stram/stream/OiOStream.java |  4 +-
 .../stram/CustomControlTupleTest.java   |  8 ++--
 15 files changed, 85 insertions(+), 83 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/apex-core/blob/d6f17f23/api/src/main/java/com/datatorrent/api/ControlTupleEnabledSink.java
--
diff --git a/api/src/main/java/com/datatorrent/api/ControlTupleEnabledSink.java 
b/api/src/main/java/com/datatorrent/api/ControlTupleEnabledSink.java
index e27003d..eeb952a 100644
--- a/api/src/main/java/com/datatorrent/api/ControlTupleEnabledSink.java
+++ b/api/src/main/java/com/datatorrent/api/ControlTupleEnabledSink.java
@@ -18,7 +18,7 @@
  */
 package com.datatorrent.api;
 
-import org.apache.apex.api.UserDefinedControlTuple;
+import org.apache.apex.api.operator.ControlTuple;
 import org.apache.hadoop.classification.InterfaceStability;
 
 /**
@@ -35,7 +35,7 @@ public interface ControlTupleEnabledSink extends Sink
 }
 
 @Override
-public boolean putControl(UserDefinedControlTuple payload)
+public boolean putControl(ControlTuple payload)
 {
   return true;
 }
@@ -52,5 +52,5 @@ public interface ControlTupleEnabledSink extends Sink
*
* @param payload the control tuple payload
*/
-  public boolean putControl(UserDefinedControlTuple payload);
+  public boolean putControl(ControlTuple payload);
 }

http://git-wip-us.apache.org/repos/asf/apex-core/blob/d6f17f23/api/src/main/java/org/apache/apex/api/ControlAwareDefaultInputPort.java
--
diff --git 
a/api/src/main/java/org/apache/apex/api/ControlAwareDefaultInputPort.java 
b/api/src/main/java/org/apache/apex/api/ControlAwareDefaultInputPort.java
index ff2b849..07e8343 100644
--- a/api/src/main/java/org/apache/apex/api/ControlAwareDefaultInputPort.java
+++ b/api/src/main/java/org/apache/apex/api/ControlAwareDefaultInputPort.java
@@ -18,6 +18,7 @@
  */
 package org.apache.apex.api;
 
+import org.apache.apex.api.operator.ControlTuple;
 import org.apache.hadoop.classification.InterfaceStability;
 
 import com.datatorrent.api.ControlTupleEnabledSink;
@@ -31,7 +32,7 @@ import com.datatorrent.api.DefaultInputPort;
 public abstract class ControlAwareDefaultInputPort extends 
DefaultInputPort implements ControlTupleEnabledSink
 {
   @Override
-  public boolean putControl(UserDefinedControlTuple payload)
+  public boolean putControl(ControlTuple payload)
   {
 count++;
 return processControl(payload);
@@ -42,5 +43,5 @@ public abstract class ControlAwareDefaultInputPort extends 
DefaultInputPorthttp://git-wip-us.apache.org/repos/asf/apex-core/blob/d6f17f23/api/src/main/java/org/apache/apex/api/ControlAwareDefaultOutputPort.java
--
diff --git 
a/api/src/main/java/org/apache/apex/api/ControlAwareDefaultOutputPort.java 
b/api/src/main/java/org/apache/apex/api/ControlAwareDefaultOutputPort.java
index 4a83518..77addf9 100644
--- a/api/src/main/java/org/apache/apex/api/ControlAwareDefaultOutputPort.java
+++ b/api/src/main/java/org/apache/apex/api/ControlAwareDefaultOutputPort.java
@@ -18,6 +18,7 @@
  */
 package org.apache.apex.api;
 
+import org.apache.apex.api.operator.ControlTuple;
 import

apex-core git commit: APEXCORE-674 Change access specifier of DTConfiguration.ValueEntry to private

2017-03-20 Thread vrozov 
Repository: apex-core
Updated Branches:
  refs/heads/master d6f17f23a -> 491e2e332


APEXCORE-674 Change access specifier of DTConfiguration.ValueEntry to private


Project: http://git-wip-us.apache.org/repos/asf/apex-core/repo
Commit: http://git-wip-us.apache.org/repos/asf/apex-core/commit/491e2e33
Tree: http://git-wip-us.apache.org/repos/asf/apex-core/tree/491e2e33
Diff: http://git-wip-us.apache.org/repos/asf/apex-core/diff/491e2e33

Branch: refs/heads/master
Commit: 491e2e332a223304209b7835a8ac762b112b7318
Parents: d6f17f2
Author: ajaygit158 
Authored: Sun Mar 19 15:48:50 2017 +0530
Committer: ajaygit158 
Committed: Mon Mar 20 22:38:07 2017 +0530

--
 .../stram/client/DTConfiguration.java   | 21 +---
 .../stram/client/DTConfigurationTest.java   | 35 
 2 files changed, 44 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/apex-core/blob/491e2e33/engine/src/main/java/com/datatorrent/stram/client/DTConfiguration.java
--
diff --git 
a/engine/src/main/java/com/datatorrent/stram/client/DTConfiguration.java 
b/engine/src/main/java/com/datatorrent/stram/client/DTConfiguration.java
index 1f19d71..45e7826 100644
--- a/engine/src/main/java/com/datatorrent/stram/client/DTConfiguration.java
+++ b/engine/src/main/java/com/datatorrent/stram/client/DTConfiguration.java
@@ -70,7 +70,7 @@ public class DTConfiguration implements 
Iterable>
   private final Map map = new LinkedHashMap<>();
   private static final Logger LOG = 
LoggerFactory.getLogger(DTConfiguration.class);
 
-  public static class ValueEntry
+  private static class ValueEntry
   {
 public String value;
 public boolean isFinal = false;
@@ -275,36 +275,33 @@ public class DTConfiguration implements 
Iterable>
 map.remove(key);
   }
 
-  public ValueEntry setInternal(String key, String value)
+  public void setInternal(String key, String value)
   {
-ValueEntry valueEntry;
-if (map.containsKey(key)) {
-  valueEntry = map.get(key);
+ValueEntry valueEntry = map.get(key);
+if (valueEntry != null) {
   valueEntry.value = value;
 } else {
   valueEntry = new ValueEntry();
   valueEntry.scope = isLocalKey(key) ? Scope.LOCAL : Scope.TRANSIENT;
+  valueEntry.value = value;
   map.put(key, valueEntry);
 }
-return valueEntry;
   }
 
-  public ValueEntry set(String key, String value, Scope scope, String 
description) throws ConfigException
+  public void set(String key, String value, Scope scope, String description) 
throws ConfigException
   {
-ValueEntry valueEntry;
-if (map.containsKey(key)) {
-  valueEntry = map.get(key);
+ValueEntry valueEntry = map.get(key);
+if (valueEntry != null) {
   if (valueEntry.isFinal) {
 throw new ConfigException("Cannot set final property " + key);
   }
 } else {
   valueEntry = new ValueEntry();
+  map.put(key, valueEntry);
 }
 valueEntry.value = value;
 valueEntry.description = description;
 valueEntry.scope = isLocalKey(key) ? Scope.LOCAL : scope;
-map.put(key, valueEntry);
-return valueEntry;
   }
 
   public static boolean isLocalKey(String key)

http://git-wip-us.apache.org/repos/asf/apex-core/blob/491e2e33/engine/src/test/java/com/datatorrent/stram/client/DTConfigurationTest.java
--
diff --git 
a/engine/src/test/java/com/datatorrent/stram/client/DTConfigurationTest.java 
b/engine/src/test/java/com/datatorrent/stram/client/DTConfigurationTest.java
new file mode 100644
index 000..c43c1f9
--- /dev/null
+++ b/engine/src/test/java/com/datatorrent/stram/client/DTConfigurationTest.java
@@ -0,0 +1,35 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package com.datatorrent.stram.client;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+public class DTConfigurationTest
+{
+  @Test
+