[GitHub] joshfischer1108 opened a new pull request #2904: Test to verify gitbox change

[GitBox]

joshfischer1108 opened a new pull request #2904: Test to verify gitbox change
URL: https://github.com/apache/incubator-heron/pull/2904
 
 
   Do not merge me.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] joshfischer1108 commented on issue #2904: Test to verify gitbox change

[GitBox]

joshfischer1108 commented on issue #2904: Test to verify gitbox change
URL: https://github.com/apache/incubator-heron/pull/2904#issuecomment-390435422
 
 
   Test comment


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] joshfischer1108 closed pull request #2904: Test to verify gitbox change

[GitBox]

joshfischer1108 closed pull request #2904: Test to verify gitbox change
URL: https://github.com/apache/incubator-heron/pull/2904
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/README.md b/README.md
index 3e21c5ee5f..6813eb54c5 100644
--- a/README.md
+++ b/README.md
@@ -1,7 +1,7 @@
 [![Build 
Status](https://travis-ci.org/apache/incubator-heron.svg?&branch=master)](https://travis-ci.org/apache/incubator-heron)
 
 ![logo](website/static/img/HeronTextLogo.png)
-
+# This is test, do not merge me
 Heron is realtime analytics platform developed by Twitter. It is the direct
 successor of [Apache Storm](http://storm.apache.org), built to be backwards
 compatible with Storm's [topology 
API](http://storm.apache.org/releases/current/Tutorial.html#topologies)


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] jerrypeng commented on issue #2692: Making emit, ack, and fail thread safe

[GitBox]

jerrypeng commented on issue #2692: Making emit, ack, and fail thread safe
URL: https://github.com/apache/incubator-heron/pull/2692#issuecomment-390544343
 
 
   @maosongfu can you approve? or do you have anymore concerns?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ajorgensen commented on issue #2900: Fix Potential Double Delete Bug

[GitBox]

ajorgensen commented on issue #2900: Fix Potential Double Delete Bug
URL: https://github.com/apache/incubator-heron/pull/2900#issuecomment-390659395
 
 
   @nlu90 fixed. Is there a bazel command just run checkstyle?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kramasamy commented on issue #2900: Fix Potential Double Delete Bug

[GitBox]

kramasamy commented on issue #2900: Fix Potential Double Delete Bug
URL: https://github.com/apache/incubator-heron/pull/2900#issuecomment-390660238
 
 
   @ajorgensen - here it is
   
   bazel build --config=darwin 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kramasamy commented on issue #2900: Fix Potential Double Delete Bug

[GitBox]

kramasamy commented on issue #2900: Fix Potential Double Delete Bug
URL: https://github.com/apache/incubator-heron/pull/2900#issuecomment-390660238
 
 
   @ajorgensen - here it is
   ```
   bazel build --config=darwin 
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ajorgensen commented on issue #2900: Fix Potential Double Delete Bug

[GitBox]

ajorgensen commented on issue #2900: Fix Potential Double Delete Bug
URL: https://github.com/apache/incubator-heron/pull/2900#issuecomment-390664392
 
 
   @kramasamy  Oh interesting... the checkstyles never run for me during a 
normal build. For example if I introduce the same checkstyle issue and run
   
   ` bazel build //heron/common/src/cpp/..` 
   
   It builds successfully. Its not until I run the `scripts/packages` directive 
that the checkstyle actually fires.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ajorgensen commented on issue #2900: Fix Potential Double Delete Bug

[GitBox]

ajorgensen commented on issue #2900: Fix Potential Double Delete Bug
URL: https://github.com/apache/incubator-heron/pull/2900#issuecomment-390664392
 
 
   @kramasamy  Oh interesting... the checkstyles never run for me during a 
normal build. For example if I introduce the same checkstyle issue and run
   
   ` bazel build //heron/common/src/cpp/..` 
   
   It builds successfully. Its not until I run the `scripts/packages` directive 
that the checkstyle actually fires. Or am I misunderstanding and there are 
specific checkstyle targets that can be run?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nwangtw commented on issue #2692: Making emit, ack, and fail thread safe

[GitBox]

nwangtw commented on issue #2692: Making emit, ack, and fail thread safe
URL: https://github.com/apache/incubator-heron/pull/2692#issuecomment-390711888
 
 
   I think maosong's concern is still valid. For stateful process, we have to 
make sure the checkpoint are made after all user threads have done their acks.  
Which is why Neng said above "But I think we need to mention the thread-safe 
acking to users very carefully, ensures them that EFFECTLY-ONCE might be broken 
if they don't implement preSave carefully."
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] jerrypeng commented on issue #2692: Making emit, ack, and fail thread safe

[GitBox]

jerrypeng commented on issue #2692: Making emit, ack, and fail thread safe
URL: https://github.com/apache/incubator-heron/pull/2692#issuecomment-390730675
 
 
   @nwangtw I agree that effectively-once semantics might not be honored if 
users decide process tuples asynchronously, but it is possible if the user uses 
the presave function as a trigger to create synchrony among user threads.  I 
think it is matter of education and documentation to educate users on this, but 
I think this PR is good go.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nwangtw commented on issue #2692: Making emit, ack, and fail thread safe

[GitBox]

nwangtw commented on issue #2692: Making emit, ack, and fail thread safe
URL: https://github.com/apache/incubator-heron/pull/2692#issuecomment-390739147
 
 
   @jerrypeng agreed. The concern is that users CAN use the function in the 
wrong way. Avoiding it in code would be nice; protecting the case might be 
helpful and acceptable too; otherwise very clear documentation and education is 
needed (we should have it anyway though). I think maosong prefers to have one 
of the first 2 options available.
   
   Maybe we should have an example code for presave() function if we want to 
cover stateful process?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] huijunwu opened a new pull request #2905: log the object when it cannot be serialized when emitting tuples

[GitBox]

huijunwu opened a new pull request #2905: log the object when it cannot be 
serialized when emitting tuples
URL: https://github.com/apache/incubator-heron/pull/2905
 
 
   We saw some topology may encounter the NotSerializableException and the 
heron-instance crashes. But the user may not be able to analyze why his tuple 
has NotSerializableException. This PR logs the tuple content to make the 
diagnosis easy.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nwangtw commented on a change in pull request #2905: log the object when it cannot be serialized when emitting tuples

[GitBox]

nwangtw commented on a change in pull request #2905: log the object when it 
cannot be serialized when emitting tuples
URL: https://github.com/apache/incubator-heron/pull/2905#discussion_r189724900
 
 

 ##
 File path: 
heron/api/src/java/org/apache/heron/api/serializer/JavaSerializer.java
 ##
 @@ -40,7 +40,7 @@ public void initialize(Map config) {
   oos.writeObject(object);
   oos.flush();
 } catch (IOException e) {
-  throw new RuntimeException(e);
+  throw new RuntimeException("NotSerializable object is: " + 
object.toString(), e);
 
 Review comment:
   Suggest: "Failed to serialize object: " + object.toString()


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] huijunw commented on a change in pull request #2905: log the object when it cannot be serialized when emitting tuples

[GitBox]

huijunw commented on a change in pull request #2905: log the object when it 
cannot be serialized when emitting tuples
URL: https://github.com/apache/incubator-heron/pull/2905#discussion_r189730038
 
 

 ##
 File path: 
heron/api/src/java/org/apache/heron/api/serializer/JavaSerializer.java
 ##
 @@ -40,7 +40,7 @@ public void initialize(Map config) {
   oos.writeObject(object);
   oos.flush();
 } catch (IOException e) {
-  throw new RuntimeException(e);
+  throw new RuntimeException("NotSerializable object is: " + 
object.toString(), e);
 
 Review comment:
   updated


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nlu90 closed pull request #2692: Making emit, ack, and fail thread safe

[GitBox]

nlu90 closed pull request #2692: Making emit, ack, and fail thread safe
URL: https://github.com/apache/incubator-heron/pull/2692
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/heron/instance/src/java/org/apache/heron/instance/AbstractOutputCollector.java
 
b/heron/instance/src/java/org/apache/heron/instance/AbstractOutputCollector.java
index 858773994f..5fb74be7ee 100644
--- 
a/heron/instance/src/java/org/apache/heron/instance/AbstractOutputCollector.java
+++ 
b/heron/instance/src/java/org/apache/heron/instance/AbstractOutputCollector.java
@@ -17,6 +17,7 @@
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
+import java.util.concurrent.locks.ReentrantLock;
 
 import com.google.protobuf.ByteString;
 import com.google.protobuf.Message;
@@ -40,6 +41,7 @@
   private long totalTuplesEmitted;
   private long totalBytesEmitted;
   private PhysicalPlanHelper helper;
+  public final ReentrantLock lock = new ReentrantLock();
 
   /**
* The SuppressWarnings is only until TOPOLOGY_ENABLE_ACKING exists.
@@ -73,7 +75,7 @@ public AbstractOutputCollector(IPluggableSerializer 
serializer,
   }
 }
 
-this.outputter = new OutgoingTupleCollection(helper, streamOutQueue);
+this.outputter = new OutgoingTupleCollection(helper, streamOutQueue, lock);
   }
 
   public void updatePhysicalPlanHelper(PhysicalPlanHelper physicalPlanHelper) {
diff --git 
a/heron/instance/src/java/org/apache/heron/instance/OutgoingTupleCollection.java
 
b/heron/instance/src/java/org/apache/heron/instance/OutgoingTupleCollection.java
index a346c4b95c..ae9e31a4fb 100644
--- 
a/heron/instance/src/java/org/apache/heron/instance/OutgoingTupleCollection.java
+++ 
b/heron/instance/src/java/org/apache/heron/instance/OutgoingTupleCollection.java
@@ -15,6 +15,8 @@
 package org.apache.heron.instance;
 
 import java.io.Serializable;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.ReentrantLock;
 
 import com.google.protobuf.ByteString;
 import com.google.protobuf.Message;
@@ -55,14 +57,17 @@
   private HeronTuples.HeronControlTupleSet.Builder currentControlTuple;
 
   // Total data emitted in bytes for the entire life
-  private long totalDataEmittedInBytes;
+  private AtomicLong totalDataEmittedInBytes = new AtomicLong();
 
   // Current size in bytes for data types to pack into the HeronTupleSet
   private long currentDataTupleSizeInBytes;
 
+  private final ReentrantLock lock;
+
   public OutgoingTupleCollection(
   PhysicalPlanHelper helper,
-  Communicator outQueue) {
+  Communicator outQueue,
+  ReentrantLock lock) {
 this.outQueue = outQueue;
 this.helper = helper;
 SystemConfig systemConfig =
@@ -72,17 +77,23 @@ public OutgoingTupleCollection(
 
SerializeDeSerializeHelper.getSerializer(helper.getTopologyContext().getTopologyConfig());
 
 // Initialize the values in constructor
-this.totalDataEmittedInBytes = 0;
+this.totalDataEmittedInBytes.set(0);
 this.currentDataTupleSizeInBytes = 0;
 
 // Read the config values
 this.dataTupleSetCapacity = systemConfig.getInstanceSetDataTupleCapacity();
 this.maxDataTupleSize = systemConfig.getInstanceSetDataTupleSize();
 this.controlTupleSetCapacity = 
systemConfig.getInstanceSetControlTupleCapacity();
+this.lock = lock;
   }
 
   public void sendOutTuples() {
-flushRemaining();
+lock.lock();
+try {
+  flushRemaining();
+} finally {
+  lock.unlock();
+}
   }
 
   /**
@@ -92,71 +103,93 @@ public void sendOutTuples() {
*/
   public void sendOutState(State state,
String checkpointId) {
-// flush all the current data before sending the state
-flushRemaining();
-
-// Serialize the state
-byte[] serializedState = serializer.serialize(state);
-
-// Construct the instance state checkpoint
-CheckpointManager.InstanceStateCheckpoint instanceState =
-CheckpointManager.InstanceStateCheckpoint.newBuilder()
-  .setCheckpointId(checkpointId)
-  .setState(ByteString.copyFrom(serializedState))
-  .build();
-
-CheckpointManager.StoreInstanceStateCheckpoint storeRequest =
-CheckpointManager.StoreInstanceStateCheckpoint.newBuilder()
-.setState(instanceState)
-.build();
-
-// Put the checkpoint to out stream queue
-outQueue.offer(storeRequest);
+lock.lock();
+try {
+  // flush all the current data before sending the state
+  flushRemaining();
+
+  // Serialize the state
+  byte[] serializedState = serializer.serialize(state);
+
+  // Construct the instance state checkpoint
+  CheckpointManager.InstanceStateCheckpoint instanceState =
+  CheckpointManager.I

[GitHub] huijunwu closed pull request #2905: log the object when it cannot be serialized when emitting tuples

[GitBox]

huijunwu closed pull request #2905: log the object when it cannot be serialized 
when emitting tuples
URL: https://github.com/apache/incubator-heron/pull/2905
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] westurner opened a new issue #2906: Website: text extends beyond boxes

[GitBox]

westurner opened a new issue #2906: Website: text extends beyond boxes
URL: https://github.com/apache/incubator-heron/issues/2906
 
 
   With a 5" phone in portrait mode, the text in the boxes under the "Why 
heron?" heading extends beyond the bottom border of the boxes.
   https://apache.github.io/incubator-heron/
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] westurner commented on issue #2906: Website: text extends beyond boxes

[GitBox]

westurner commented on issue #2906: Website: text extends beyond boxes
URL: 
https://github.com/apache/incubator-heron/issues/2906#issuecomment-391217102
 
 
   
![screenshot_20180523-002832](https://user-images.githubusercontent.com/50891/40403586-c4620f20-5e20-11e8-8c5c-0c8865957f89.png)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] huijunw opened a new issue #2907: stuck stmgr due to zk client destructor

[GitBox]

huijunw opened a new issue #2907: stuck stmgr due to zk client destructor
URL: https://github.com/apache/incubator-heron/issues/2907
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] huijunw commented on issue #2907: stuck stmgr due to zk client destructor

[GitBox]

huijunw commented on issue #2907: stuck stmgr due to zk client destructor
URL: 
https://github.com/apache/incubator-heron/issues/2907#issuecomment-391482300
 
 
   two questions:
   1. why is zookeeper not single thread ? `libzookeeper_mt` vs 
`libzookeeper_st`
   2. if multi thread zk client is expected, how to solve the race condition? 
blind guess: delete piper after zookeeper_close() 
https://github.com/apache/incubator-heron/blob/0.17.8/heron/common/src/cpp/zookeeper/zkclient.cpp#L146


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] huijunw commented on issue #2907: stuck stmgr due to zk client destructor

[GitBox]

huijunw commented on issue #2907: stuck stmgr due to zk client destructor
URL: 
https://github.com/apache/incubator-heron/issues/2907#issuecomment-391489175
 
 
   replace https://github.com/apache/incubator-heron/issues/2765


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] huijunw commented on issue #2907: stuck stmgr due to zk client destructor

[GitBox]

huijunw commented on issue #2907: stuck stmgr due to zk client destructor
URL: 
https://github.com/apache/incubator-heron/issues/2907#issuecomment-391510536
 
 
   For the thread 0x7f616cc62700:
   It called the GetCompletionWatcher(), when a getting-zk-node operation is 
done. In the watcher, ZkActionCb-> ExecuteInEventLoop-> enqueue-> notify_one-> 
__lll_lock_wait(), stuck.
   
   For the thread main 0x7f616ecae780:
   A zk session expired, GlobalWatchEventHandler was called -> ~ZKClient() -> 
first delete piper, then zookeeper_close() -> join thread, stuck
   
   Our theory is: two events(session_expire and get_zk_node) happened, and each 
was handled in a thread. The main thread handled session_expire, while the 
other thread handled get_zk_node_done. The session_expire watcher in main wait 
for the other thread to join, while the other thread wait for a lock which was 
deleted by  ~ZKClient() in main thread.
   
   If multi thread is intended, proposed solution is:
   reorder the delete_piper and close_zk_client
   ```
 delete piper_;
 zookeeper_close(zk_handle_);
   ```
   
https://github.com/apache/incubator-heron/blob/0.17.8/heron/common/src/cpp/zookeeper/zkclient.cpp#L146
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] huijunw commented on issue #2907: stuck stmgr due to zk client destructor

[GitBox]

huijunw commented on issue #2907: stuck stmgr due to zk client destructor
URL: 
https://github.com/apache/incubator-heron/issues/2907#issuecomment-391510536
 
 
   For the thread 0x7f616cc62700:
   It called the GetCompletionWatcher(), when a getting-zk-node operation is 
done. In the watcher, ZkActionCb-> ExecuteInEventLoop-> enqueue-> notify_one-> 
__lll_lock_wait(), stuck.
   
   For the thread main 0x7f616ecae780:
   A zk session expired, GlobalWatchEventHandler was called -> ~ZKClient() -> 
first delete piper, then zookeeper_close() -> join thread, stuck
   
   Our theory is: two events(session_expire and get_zk_node) happened, and each 
was handled in a thread. The main thread handled session_expire, while the 
other thread handled get_zk_node_done. The session_expire watcher in main wait 
for the other thread to join, while the other thread wait for a lock which was 
deleted in piper_ by  ~ZKClient() in main thread.
   
   If multi thread is intended, proposed solution is:
   reorder the delete_piper and close_zk_client
   ```
 delete piper_;
 zookeeper_close(zk_handle_);
   ```
   
https://github.com/apache/incubator-heron/blob/0.17.8/heron/common/src/cpp/zookeeper/zkclient.cpp#L146
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] huijunwu opened a new pull request #2908: reorder operations in zk_client destructor

[GitBox]

huijunwu opened a new pull request #2908: reorder operations in zk_client 
destructor
URL: https://github.com/apache/incubator-heron/pull/2908
 
 
   fix https://github.com/apache/incubator-heron/issues/2907


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nlu90 commented on issue #2907: stuck stmgr due to zk client destructor

[GitBox]

nlu90 commented on issue #2907: stuck stmgr due to zk client destructor
URL: 
https://github.com/apache/incubator-heron/issues/2907#issuecomment-391530890
 
 
   The comment 
[here](https://github.com/apache/incubator-heron/blob/master/heron/statemgrs/src/cpp/statemgr/heron-zkstatemgr.cpp#L358)
 claims it's single thread mode. But from the dump we found there're actually 2 
threads. So we need to update the comments to correctly reflect what the code 
does.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nlu90 commented on issue #2907: stuck stmgr due to zk client destructor

[GitBox]

nlu90 commented on issue #2907: stuck stmgr due to zk client destructor
URL: 
https://github.com/apache/incubator-heron/issues/2907#issuecomment-391530890
 
 
   
[here](https://stackoverflow.com/questions/8248458/pthread-cond-signal-causing-deadlock)
 is some discussion about the `__lll_lock_wait ` error.
   
   The comment 
[here](https://github.com/apache/incubator-heron/blob/master/heron/statemgrs/src/cpp/statemgr/heron-zkstatemgr.cpp#L358)
 claims it's single thread mode. But from the dump we found there're actually 2 
threads. So we need to update the comments to correctly reflect what the code 
does.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] huijunw commented on issue #2907: stuck stmgr due to zk client destructor

[GitBox]

huijunw commented on issue #2907: stuck stmgr due to zk client destructor
URL: 
https://github.com/apache/incubator-heron/issues/2907#issuecomment-391546584
 
 
   @srkukarni why is zookeeper not single thread ? libzookeeper_mt vs 
libzookeeper_st
   If I remember correctly, the stmgr should be single thread and the zkclient 
should be in the same main thread. What we observed is that zk client is multi 
thread. I would like to know if this is intended, and shall we make zk client 
share the same single main thread


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] maosongfu commented on issue #2908: reorder operations in zk_client destructor

[GitBox]

maosongfu commented on issue #2908: reorder operations in zk_client destructor
URL: https://github.com/apache/incubator-heron/pull/2908#issuecomment-391549027
 
 
   Had a discussion with Huijun, and met some new concerns. Hold it for now 
until we better understand it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] huijunw commented on issue #2907: stuck stmgr due to zk client destructor

[GitBox]

huijunw commented on issue #2907: stuck stmgr due to zk client destructor
URL: 
https://github.com/apache/incubator-heron/issues/2907#issuecomment-391554957
 
 
   @srkukarni another thought: for the pcqueue, 
https://github.com/apache/incubator-heron/blob/0.17.8/heron/common/src/cpp/threads/pcqueue.h#L36,
 when the pcqueue is destructed, the mutex and cond_ behavior is undefined. but 
pcqueue.enqueue is called after ~PCQueue(). what do you think?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] mjschmidt commented on issue #2667: Restore container's ability to run locally (or create new dev container with heron client in it so new heron container can stay small)

[GitBox]

mjschmidt commented on issue #2667: Restore container's ability to run locally 
(or create new dev container with heron client in it so new heron container can 
stay small)
URL: 
https://github.com/apache/incubator-heron/issues/2667#issuecomment-392139782
 
 
   This can be closed as resuloved :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] mjschmidt closed issue #2667: Restore container's ability to run locally (or create new dev container with heron client in it so new heron container can stay small)

[GitBox]

mjschmidt closed issue #2667: Restore container's ability to run locally (or 
create new dev container with heron client in it so new heron container can 
stay small)
URL: https://github.com/apache/incubator-heron/issues/2667
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] mjschmidt opened a new issue #2909: Add ack and fail to Streamlet API

[GitBox]

mjschmidt opened a new issue #2909: Add ack and fail to Streamlet API
URL: https://github.com/apache/incubator-heron/issues/2909
 
 
The default behavior for heron streamlet is not to ack/fail messages. Upon 
further inspection we realized this is not only the default, but the only 
option for Streamlet as it only uses emit, which will not ack and fail 
messages. This currently isn't configurable.
   
   The code can be seen here: 
   
https://github.com/apache/incubator-heron/blob/6029c28943b7ad85be6fa175d20e0e908fb199b6/heron/api/src/java/org/apache/heron/streamlet/impl/sources/ComplexSource.java#L60
   
   What steps would be required to add this functionality? This would aid 
greatly in us transitioning to Streamlet.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] jerrypeng commented on issue #2909: Add ack and fail to Streamlet API

[GitBox]

jerrypeng commented on issue #2909: Add ack and fail to Streamlet API
URL: 
https://github.com/apache/incubator-heron/issues/2909#issuecomment-392153800
 
 
   @mjschmidt so one of the design choices we made for the streamlet API was to 
move away from the concept of spouts/bolts towards the concepts of sources and 
sinks to better support state and effectively once processing provided 
chandy/lamport style state snapshotting.  Acking was more natural spout and 
bolt APIs where managed state by heron isn't available while not so much for 
the functional style APIs used in Streamlets Acking also doesn't gel well with 
the state implementation since state snapshotting is provided by chandy/lamport 
global snapshotting. Perhaps @srkukarni can also chime in.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nwangtw commented on issue #2908: reorder operations in zk_client destructor

[GitBox]

nwangtw commented on issue #2908: reorder operations in zk_client destructor
URL: https://github.com/apache/incubator-heron/pull/2908#issuecomment-392214519
 
 
   Discussed offline. Update:
   
   The fix is valid and helpful in the current implementation. The questions 
are about the implementation of queue and if we can handle zk events in a 
simpler way. So we agreed to submit the fix which could improve heron 
reliability, and then discuss about the zk client implementation in future.
   
   In conclusion, we have a green light to merge~
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] huijunwu closed issue #2907: stuck stmgr due to zk client destructor

[GitBox]

huijunwu closed issue #2907: stuck stmgr due to zk client destructor
URL: https://github.com/apache/incubator-heron/issues/2907
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] huijunwu closed pull request #2908: reorder operations in zk_client destructor

[GitBox]

huijunwu closed pull request #2908: reorder operations in zk_client destructor
URL: https://github.com/apache/incubator-heron/pull/2908
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nlu90 opened a new pull request #2910: make LocalFSStorage clean checkpoints before store

[GitBox]

nlu90 opened a new pull request #2910: make LocalFSStorage clean checkpoints 
before store
URL: https://github.com/apache/incubator-heron/pull/2910
 
 
   The current implementation of LocalFSStorage would cause unlimited 
checkpoints to be saved on local disk and thus out of disk space problem.
   
   This PR fixes the problem by adding a limit of the number of checkpoints 
saved on local disk and clean unnecessary checkpoints every time before trying 
to store a new checkpoint. The most recent N checkpoints will be saved and any 
older checkpoints will be deleted.
   
   Tested on local mac and worked as expected.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nwangtw commented on a change in pull request #2910: make LocalFSStorage clean checkpoints before store

[GitBox]

nwangtw commented on a change in pull request #2910: make LocalFSStorage clean 
checkpoints before store
URL: https://github.com/apache/incubator-heron/pull/2910#discussion_r191931212
 
 

 ##
 File path: 
heron/statefulstorages/src/java/org/apache/heron/statefulstorage/localfs/LocalFileSystemStorage.java
 ##
 @@ -134,6 +146,23 @@ public void dispose(String topologyName, String 
oldestCheckpointPreserved,
 }
   }
 
+  private void cleanCheckpoints(String path, int remaining) throws 
StatefulStorageException {
+if (FileUtils.isDirectoryExists(path) && FileUtils.hasChildren(path)) {
+  File[] children = new File(path).listFiles();
 
 Review comment:
   suggest to keep it consistent with dispose(), which uses:
   
   String[] names = new File(topologyCheckpointRoot).list();
   
   Sorting might be cheaper this way as well.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nwangtw commented on a change in pull request #2910: make LocalFSStorage clean checkpoints before store

[GitBox]

nwangtw commented on a change in pull request #2910: make LocalFSStorage clean 
checkpoints before store
URL: https://github.com/apache/incubator-heron/pull/2910#discussion_r191931672
 
 

 ##
 File path: 
heron/statefulstorages/src/java/org/apache/heron/statefulstorage/localfs/LocalFileSystemStorage.java
 ##
 @@ -134,6 +146,23 @@ public void dispose(String topologyName, String 
oldestCheckpointPreserved,
 }
   }
 
+  private void cleanCheckpoints(String path, int remaining) throws 
StatefulStorageException {
+if (FileUtils.isDirectoryExists(path) && FileUtils.hasChildren(path)) {
+  File[] children = new File(path).listFiles();
+  Arrays.sort(children, Comparator.reverseOrder());
 
 Review comment:
   I would suggest to break the logic into two steps:
   1. generate the toDelete list.
   2. for each one in the list, deleteDir().
   
   The benefit is that you can write a unit test for step 1 easily to make sure 
the order is correct.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nwangtw commented on a change in pull request #2910: make LocalFSStorage clean checkpoints before store

[GitBox]

nwangtw commented on a change in pull request #2910: make LocalFSStorage clean 
checkpoints before store
URL: https://github.com/apache/incubator-heron/pull/2910#discussion_r191929947
 
 

 ##
 File path: 
heron/statefulstorages/src/java/org/apache/heron/statefulstorage/localfs/LocalFileSystemStorage.java
 ##
 @@ -32,20 +34,25 @@
 import org.apache.heron.spi.statefulstorage.IStatefulStorage;
 import org.apache.heron.spi.statefulstorage.StatefulStorageException;
 
-public class LocalFileSystemStorage implements IStatefulStorage {
+public class  LocalFileSystemStorage implements IStatefulStorage {
   private static final Logger LOG = 
Logger.getLogger(LocalFileSystemStorage.class.getName());
 
   private static final String ROOT_PATH_KEY = 
"heron.statefulstorage.localfs.root.path";
+  private static final String MAX_CHECKPOINTS_KEY = 
"heron.statefulstorage.localfs.max.checkpoints";
 
   private String checkpointRootPath;
+  private int maxCheckpoints;
 
   @Override
   public void init(Map conf) throws StatefulStorageException {
 checkpointRootPath = (String) conf.get(ROOT_PATH_KEY);
+maxCheckpoints = (int) conf.get(MAX_CHECKPOINTS_KEY);
 
 Review comment:
   could be helpful to have a default value?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nwangtw commented on a change in pull request #2910: make LocalFSStorage clean checkpoints before store

[GitBox]

nwangtw commented on a change in pull request #2910: make LocalFSStorage clean 
checkpoints before store
URL: https://github.com/apache/incubator-heron/pull/2910#discussion_r191932377
 
 

 ##
 File path: 
heron/statefulstorages/src/java/org/apache/heron/statefulstorage/localfs/LocalFileSystemStorage.java
 ##
 @@ -134,6 +146,23 @@ public void dispose(String topologyName, String 
oldestCheckpointPreserved,
 }
   }
 
+  private void cleanCheckpoints(String path, int remaining) throws 
StatefulStorageException {
+if (FileUtils.isDirectoryExists(path) && FileUtils.hasChildren(path)) {
+  File[] children = new File(path).listFiles();
+  Arrays.sort(children, Comparator.reverseOrder());
+
+  // only keep the latest N remaining files, delete others
+  for (int i = remaining; i < children.length; i++) {
 
 Review comment:
   if we do:
   for (int i = 0; i < children.length - remaining; ++i) {
   }
   
   we dont need Comparator, and the sorting might be slightly faster because 
the list might be likely to be in the same order already.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nlu90 commented on a change in pull request #2910: make LocalFSStorage clean checkpoints before store

[GitBox]

nlu90 commented on a change in pull request #2910: make LocalFSStorage clean 
checkpoints before store
URL: https://github.com/apache/incubator-heron/pull/2910#discussion_r191938042
 
 

 ##
 File path: 
heron/statefulstorages/src/java/org/apache/heron/statefulstorage/localfs/LocalFileSystemStorage.java
 ##
 @@ -134,6 +146,23 @@ public void dispose(String topologyName, String 
oldestCheckpointPreserved,
 }
   }
 
+  private void cleanCheckpoints(String path, int remaining) throws 
StatefulStorageException {
+if (FileUtils.isDirectoryExists(path) && FileUtils.hasChildren(path)) {
+  File[] children = new File(path).listFiles();
+  Arrays.sort(children, Comparator.reverseOrder());
+
+  // only keep the latest N remaining files, delete others
+  for (int i = remaining; i < children.length; i++) {
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nlu90 commented on a change in pull request #2910: make LocalFSStorage clean checkpoints before store

[GitBox]

nlu90 commented on a change in pull request #2910: make LocalFSStorage clean 
checkpoints before store
URL: https://github.com/apache/incubator-heron/pull/2910#discussion_r191938012
 
 

 ##
 File path: 
heron/statefulstorages/src/java/org/apache/heron/statefulstorage/localfs/LocalFileSystemStorage.java
 ##
 @@ -134,6 +146,23 @@ public void dispose(String topologyName, String 
oldestCheckpointPreserved,
 }
   }
 
+  private void cleanCheckpoints(String path, int remaining) throws 
StatefulStorageException {
+if (FileUtils.isDirectoryExists(path) && FileUtils.hasChildren(path)) {
+  File[] children = new File(path).listFiles();
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nlu90 commented on a change in pull request #2910: make LocalFSStorage clean checkpoints before store

[GitBox]

nlu90 commented on a change in pull request #2910: make LocalFSStorage clean 
checkpoints before store
URL: https://github.com/apache/incubator-heron/pull/2910#discussion_r191938652
 
 

 ##
 File path: 
heron/statefulstorages/src/java/org/apache/heron/statefulstorage/localfs/LocalFileSystemStorage.java
 ##
 @@ -32,20 +34,25 @@
 import org.apache.heron.spi.statefulstorage.IStatefulStorage;
 import org.apache.heron.spi.statefulstorage.StatefulStorageException;
 
-public class LocalFileSystemStorage implements IStatefulStorage {
+public class  LocalFileSystemStorage implements IStatefulStorage {
   private static final Logger LOG = 
Logger.getLogger(LocalFileSystemStorage.class.getName());
 
   private static final String ROOT_PATH_KEY = 
"heron.statefulstorage.localfs.root.path";
+  private static final String MAX_CHECKPOINTS_KEY = 
"heron.statefulstorage.localfs.max.checkpoints";
 
   private String checkpointRootPath;
+  private int maxCheckpoints;
 
   @Override
   public void init(Map conf) throws StatefulStorageException {
 checkpointRootPath = (String) conf.get(ROOT_PATH_KEY);
+maxCheckpoints = (int) conf.get(MAX_CHECKPOINTS_KEY);
 
 Review comment:
   Providing a default value in code might cause confusion. I prefer to have 
this value comes from conf file. It's cleaner and easier to reason.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kramasamy commented on issue #2910: make LocalFSStorage clean checkpoints before store

[GitBox]

kramasamy commented on issue #2910: make LocalFSStorage clean checkpoints 
before store
URL: https://github.com/apache/incubator-heron/pull/2910#issuecomment-393335902
 
 
   @nlu90 - Do you do background cleaning of snapshots or on demand basis? It 
might be easier to do a background cleaning.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nlu90 commented on issue #2910: make LocalFSStorage clean checkpoints before store

[GitBox]

nlu90 commented on issue #2910: make LocalFSStorage clean checkpoints before 
store
URL: https://github.com/apache/incubator-heron/pull/2910#issuecomment-393337051
 
 
   @kramasamy 
   1. On demand basis cleaning is not working in localfsstorage because the 
checkpoints are stored locally and they can't receive the delete requests from 
tmaser by design.
   
   2. What do you mean by "backgroud cleaning"? Spawn another thread and keeps 
monitoring # of checkpoints? This sounds too overkill.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kramasamy commented on issue #2910: make LocalFSStorage clean checkpoints before store

[GitBox]

kramasamy commented on issue #2910: make LocalFSStorage clean checkpoints 
before store
URL: https://github.com/apache/incubator-heron/pull/2910#issuecomment-393339706
 
 
   background cleaning might be easier in the case of storage system like 
hadoop - which takes amount of time to delete several files and reclaim. 
Furthermore, it makes the checkpointing at tmaster immediate rather than 
waiting for cleanup to finish. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nlu90 commented on issue #2910: make LocalFSStorage clean checkpoints before store

[GitBox]

nlu90 commented on issue #2910: make LocalFSStorage clean checkpoints before 
store
URL: https://github.com/apache/incubator-heron/pull/2910#issuecomment-393340979
 
 
   @kramasamy This change only needed by local filesystem storage. For other 
storage systems, no change is needed and the original design works fine.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kramasamy commented on issue #2910: make LocalFSStorage clean checkpoints before store

[GitBox]

kramasamy commented on issue #2910: make LocalFSStorage clean checkpoints 
before store
URL: https://github.com/apache/incubator-heron/pull/2910#issuecomment-393341202
 
 
   Ok got it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nlu90 commented on a change in pull request #2910: make LocalFSStorage clean checkpoints before store

[GitBox]

nlu90 commented on a change in pull request #2910: make LocalFSStorage clean 
checkpoints before store
URL: https://github.com/apache/incubator-heron/pull/2910#discussion_r191938652
 
 

 ##
 File path: 
heron/statefulstorages/src/java/org/apache/heron/statefulstorage/localfs/LocalFileSystemStorage.java
 ##
 @@ -32,20 +34,25 @@
 import org.apache.heron.spi.statefulstorage.IStatefulStorage;
 import org.apache.heron.spi.statefulstorage.StatefulStorageException;
 
-public class LocalFileSystemStorage implements IStatefulStorage {
+public class  LocalFileSystemStorage implements IStatefulStorage {
   private static final Logger LOG = 
Logger.getLogger(LocalFileSystemStorage.class.getName());
 
   private static final String ROOT_PATH_KEY = 
"heron.statefulstorage.localfs.root.path";
+  private static final String MAX_CHECKPOINTS_KEY = 
"heron.statefulstorage.localfs.max.checkpoints";
 
   private String checkpointRootPath;
+  private int maxCheckpoints;
 
   @Override
   public void init(Map conf) throws StatefulStorageException {
 checkpointRootPath = (String) conf.get(ROOT_PATH_KEY);
+maxCheckpoints = (int) conf.get(MAX_CHECKPOINTS_KEY);
 
 Review comment:
   Providing a default value in code might cause confusion. I prefer to have 
this value comes from conf file. It's cleaner and easier to reason.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nlu90 commented on a change in pull request #2910: make LocalFSStorage clean checkpoints before store

[GitBox]

nlu90 commented on a change in pull request #2910: make LocalFSStorage clean 
checkpoints before store
URL: https://github.com/apache/incubator-heron/pull/2910#discussion_r191948936
 
 

 ##
 File path: 
heron/statefulstorages/src/java/org/apache/heron/statefulstorage/localfs/LocalFileSystemStorage.java
 ##
 @@ -32,20 +34,25 @@
 import org.apache.heron.spi.statefulstorage.IStatefulStorage;
 import org.apache.heron.spi.statefulstorage.StatefulStorageException;
 
-public class LocalFileSystemStorage implements IStatefulStorage {
+public class  LocalFileSystemStorage implements IStatefulStorage {
   private static final Logger LOG = 
Logger.getLogger(LocalFileSystemStorage.class.getName());
 
   private static final String ROOT_PATH_KEY = 
"heron.statefulstorage.localfs.root.path";
+  private static final String MAX_CHECKPOINTS_KEY = 
"heron.statefulstorage.localfs.max.checkpoints";
 
   private String checkpointRootPath;
+  private int maxCheckpoints;
 
   @Override
   public void init(Map conf) throws StatefulStorageException {
 checkpointRootPath = (String) conf.get(ROOT_PATH_KEY);
+maxCheckpoints = (int) conf.get(MAX_CHECKPOINTS_KEY);
 
 Review comment:
   Added the default value


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] huijunwu opened a new pull request #2911: refactor downloader build

[GitBox]

huijunwu opened a new pull request #2911: refactor downloader build
URL: https://github.com/apache/incubator-heron/pull/2911
 
 
   refactor downloader build. make it similar to metricsmgr build.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] huijunwu opened a new pull request #2912: refactor healthmgr build

[GitBox]

huijunwu opened a new pull request #2912: refactor healthmgr build
URL: https://github.com/apache/incubator-heron/pull/2912
 
 
   refactor healthmgr build. make it similar to metricsmgr build.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] huijunwu closed pull request #2911: refactor downloader build

[GitBox]

huijunwu closed pull request #2911: refactor downloader build
URL: https://github.com/apache/incubator-heron/pull/2911
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] huijunwu closed pull request #2912: refactor healthmgr build

[GitBox]

huijunwu closed pull request #2912: refactor healthmgr build
URL: https://github.com/apache/incubator-heron/pull/2912
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] mjschmidt commented on issue #2909: Add ack and fail to Streamlet API

[GitBox]

mjschmidt commented on issue #2909: Add ack and fail to Streamlet API
URL: 
https://github.com/apache/incubator-heron/issues/2909#issuecomment-393604027
 
 
   So lets say I as a Heron analytic developer want to guarantee that all of 
the data sent from one container is received in the next? In the topology model 
this was done through ack and fail. It is a capability that will be required 
for us to move to the Streamlet API. If we put a merge request in and have it 
as a feature that is turned off by default would you guys be open to a merge 
request? If not how can we get similar functionality from streamlet?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] mjschmidt commented on issue #2909: Add ack and fail to Streamlet API

[GitBox]

mjschmidt commented on issue #2909: Add ack and fail to Streamlet API
URL: 
https://github.com/apache/incubator-heron/issues/2909#issuecomment-393604027
 
 
   So lets say I as a Heron analytic developer want to guarantee that all of 
the data sent from one container is received in the next? In the topology model 
this was done through ack and fail. It is a capability that will be required 
for us to move to the Streamlet API. If we put a pull request in and have it as 
a feature that is turned off by default would you guys be open to a merge 
request? If not how can we get similar functionality from Streamlet?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] mjschmidt commented on issue #2909: Add ack and fail to Streamlet API

[GitBox]

mjschmidt commented on issue #2909: Add ack and fail to Streamlet API
URL: 
https://github.com/apache/incubator-heron/issues/2909#issuecomment-393604027
 
 
   @jerrypeng  So lets say I as a Heron analytic developer want to guarantee 
that all of the data sent from one container is received in the next? In the 
topology model this was done through ack and fail. It is a capability that will 
be required for us to move to the Streamlet API. If we put a pull request in 
and have it as a feature that is turned off by default would you guys be open 
to a merge request? If not how can we get similar functionality from Streamlet?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] mjschmidt commented on issue #2909: Add ack and fail to Streamlet API

[GitBox]

mjschmidt commented on issue #2909: Add ack and fail to Streamlet API
URL: 
https://github.com/apache/incubator-heron/issues/2909#issuecomment-393604027
 
 
   @jerrypeng  So lets say I as a Heron analytic developer want to guarantee 
that all of the data sent from one container is received in the next? In the 
topology model this was done through ack and fail. It is a capability that will 
be required for us to move to the Streamlet API/Heron. If we put a pull request 
in and have it as a feature that is turned off by default would you guys be 
open to a merge request? If not how can we get similar functionality from 
Streamlet?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kramasamy commented on issue #2909: Add ack and fail to Streamlet API

[GitBox]

kramasamy commented on issue #2909: Add ack and fail to Streamlet API
URL: 
https://github.com/apache/incubator-heron/issues/2909#issuecomment-393643114
 
 
   @jerrypeng @maosongfu @nwangtw - can you respond?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kramasamy commented on issue #2909: Add ack and fail to Streamlet API

[GitBox]

kramasamy commented on issue #2909: Add ack and fail to Streamlet API
URL: 
https://github.com/apache/incubator-heron/issues/2909#issuecomment-393699760
 
 
   @mjschmidt - this is due to the underlying bolt implementation that 
streamlet maps into. Currently these bolts do not ack or fail when the topology 
is set to ATLEAST_ONCE. We need to fix this. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ajorgensen closed pull request #2766: Update build-artifacts.sh script path in docs

[GitBox]

ajorgensen closed pull request #2766: Update build-artifacts.sh script path in 
docs
URL: https://github.com/apache/incubator-heron/pull/2766
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/bazel_configure.py b/bazel_configure.py
index b91efe253d..865cb6d836 100755
--- a/bazel_configure.py
+++ b/bazel_configure.py
@@ -8,7 +8,7 @@
 # When changing this script, verify that it still works by running locally
 # on a mac. Then verify the other environments by doing this:
 #
-#  cd docker
+#  cd docker/scripts
 #  ./build-artifacts.sh ubuntu15.10 0.12.0 .
 #  ./build-artifacts.sh ubuntu14.04 0.12.0 .
 #  ./build-artifacts.sh centos7 0.12.0 .
diff --git a/website/content/docs/developers/compiling/docker.md 
b/website/content/docs/developers/compiling/docker.md
index 2e11da23ff..0a85ada9f0 100644
--- a/website/content/docs/developers/compiling/docker.md
+++ b/website/content/docs/developers/compiling/docker.md
@@ -34,13 +34,13 @@ Heron provides a `build-arfifacts.sh` script for Docker 
located in the
 
 ```bash
 $ cd /path/to/heron/repo
-$ docker/build-artifacts.sh
+$ docker/scripts/build-artifacts.sh
 ```
 
 Running the script by itself will display usage information:
 
 ```
-Usage: docker/build-artifacts.sh   [source-tarball] 

+Usage: docker/scripts/build-artifacts.sh   
[source-tarball] 
 
 Platforms Supported: darwin, ubuntu14.04, ubuntu15.10, centos7
 
@@ -64,7 +64,7 @@ The following arguments are required:
 Here's an example usage:
 
 ```bash
-$ docker/build-artifacts.sh ubuntu14.04 0.12.0 ~/heron-release
+$ docker/scripts/build-artifacts.sh ubuntu14.04 0.12.0 ~/heron-release
 ```
 
 This will build a Docker container specific to Ubuntu 14.04, create a source
@@ -87,15 +87,13 @@ of the generated artifacts:
 
 ```bash
 $ ls ~/heron-release
+heron-0.12.0-ubuntu14.04.tar
+heron-0.12.0-ubuntu14.04.tar.gz
 heron-api-0.12.0-ubuntu14.04.tar.gz
-heron-client-0.12.0-ubuntu14.04.tar.gz
+heron-core-0.12.0-ubuntu14.04.tar.gz
+heron-install-0.12.0-ubuntu14.04.sh
+heron-layer-0.12.0-ubuntu14.04.tar
 heron-tools-0.12.0-ubuntu14.04.tar.gz
-heron-client-install-0.12.0-ubuntu.sh  
-heron-tools-install-0.12.0-ubuntu.sh
-heron-api-install-0.12.0-ubuntu.sh 
-heron-core-0.12.0-ubuntu.tar.gz
-```
-
 ## Contributing New Environments
 
 You'll notice that there are multiple


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ajorgensen closed pull request #2850: Add SSL support for stream manager connections

[GitBox]

ajorgensen closed pull request #2850: Add SSL support for stream manager 
connections
URL: https://github.com/apache/incubator-heron/pull/2850
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/WORKSPACE b/WORKSPACE
index 72b582f2a5..c150a6c575 100644
--- a/WORKSPACE
+++ b/WORKSPACE
@@ -837,6 +837,13 @@ new_http_archive(
 sha256 = 
"965cc5a8bb46ce4199a47e9b2c9e1cae3b137e8356ffdad6d94d3b9069b71dc2",
 )
 
+new_http_archive(
+name = "org_openssl_openssl",
+urls = ["https://www.openssl.org/source/openssl-1.1.0h.tar.gz";],
+strip_prefix = "openssl-1.1.0h",
+build_file = "third_party/openssl/openssl.BUILD",
+)
+
 new_http_archive(
 name = "org_nongnu_libunwind",
 urls = 
["http://download.savannah.nongnu.org/releases/libunwind/libunwind-1.1.tar.gz";],
diff --git a/heron/common/src/cpp/config/heron-internals-config-reader.cpp 
b/heron/common/src/cpp/config/heron-internals-config-reader.cpp
index 174dcb8d6d..19a7b60f68 100644
--- a/heron/common/src/cpp/config/heron-internals-config-reader.cpp
+++ b/heron/common/src/cpp/config/heron-internals-config-reader.cpp
@@ -292,6 +292,16 @@ int 
HeronInternalsConfigReader::GetHeronInstanceReconnectStreammgrTimes() {
   .as();
 }
 
+sp_string 
HeronInternalsConfigReader::GetHeronStreammgrEncryptionCertificatePath() {
+  return 
config_[HeronInternalsConfigVars::HERON_STREAMMGR_ENCRYPTION_CERTIFICATE_PATH]
+ .as();
+}
+
+sp_string 
HeronInternalsConfigReader::GetHeronStreammgrEncryptionPrivateKeyPath() {
+  return 
config_[HeronInternalsConfigVars::HERON_STREAMMGR_ENCRYPTION_PRIVATE_KEY_PATH]
+ .as();
+}
+
 int 
HeronInternalsConfigReader::GetHeronInstanceInternalBoltReadQueueCapacity() {
   return 
config_[HeronInternalsConfigVars::HERON_INSTANCE_INTERNAL_BOLT_READ_QUEUE_CAPACITY]
   .as();
diff --git a/heron/common/src/cpp/config/heron-internals-config-reader.h 
b/heron/common/src/cpp/config/heron-internals-config-reader.h
index d5a34a1702..06e0161876 100644
--- a/heron/common/src/cpp/config/heron-internals-config-reader.h
+++ b/heron/common/src/cpp/config/heron-internals-config-reader.h
@@ -191,6 +191,12 @@ class HeronInternalsConfigReader : public YamlFileReader {
   // Low water mark on the num in MB that can be left outstanding on a 
connection
   sp_int32 GetHeronStreammgrNetworkBackpressureLowwatermarkMb();
 
+  // Path to the ssl certificate for stream manager communication
+  sp_string GetHeronStreammgrEncryptionCertificatePath();
+
+  // Path to the private key for stream manager communication
+  sp_string GetHeronStreammgrEncryptionPrivateKeyPath();
+
   /**
   * Instance Config Getters
   **/
diff --git a/heron/common/src/cpp/config/heron-internals-config-vars.cpp 
b/heron/common/src/cpp/config/heron-internals-config-vars.cpp
index 0906eaa15f..1b1ff02d53 100644
--- a/heron/common/src/cpp/config/heron-internals-config-vars.cpp
+++ b/heron/common/src/cpp/config/heron-internals-config-vars.cpp
@@ -112,6 +112,10 @@ const sp_string 
HeronInternalsConfigVars::HERON_STREAMMGR_NETWORK_BACKPRESSURE_L
 "heron.streammgr.network.backpressure.lowwatermark.mb";
 const sp_string 
HeronInternalsConfigVars::HERON_STREAMMGR_STATEFUL_BUFFER_SIZE_MB =
 "heron.streammgr.stateful.buffer.size.mb";
+const sp_string 
HeronInternalsConfigVars::HERON_STREAMMGR_ENCRYPTION_CERTIFICATE_PATH =
+"heron.streammgr.encryption.certificate_path";
+const sp_string 
HeronInternalsConfigVars::HERON_STREAMMGR_ENCRYPTION_PRIVATE_KEY_PATH =
+"heron.streammgr.encryption.private_key_path";
 
 // heron.instance.* configs are for the instance
 const sp_string 
HeronInternalsConfigVars::HERON_INSTANCE_RECONNECT_STREAMMGR_INTERVAL_SEC =
diff --git a/heron/common/src/cpp/config/heron-internals-config-vars.h 
b/heron/common/src/cpp/config/heron-internals-config-vars.h
index 5ac37523b7..a52c92a743 100644
--- a/heron/common/src/cpp/config/heron-internals-config-vars.h
+++ b/heron/common/src/cpp/config/heron-internals-config-vars.h
@@ -181,6 +181,12 @@ class HeronInternalsConfigVars {
   // checkpoint markers to arrive before giving up
   static const sp_string HERON_STREAMMGR_STATEFUL_BUFFER_SIZE_MB;
 
+  // Path to the ssl certificate for stream manager communication
+  static const sp_string HERON_STREAMMGR_ENCRYPTION_CERTIFICATE_PATH;
+
+  // Path to the private key for stream manager communication
+  static const sp_string HERON_STREAMMGR_ENCRYPTION_PRIVATE_KEY_PATH;
+
   /**
   * HERON_INSTANCE_* configs are for the instance
   **/
diff --git a/heron/common/src/cpp/network/BUILD 
b/heron/common/src/cpp/network/BUILD
index 30c7cefecb..43bf26dab0 100644
--- a/heron/common/src/cpp/network/BUILD
+++ b/heron/common/src/cpp/network/BUILD
@@ -3,53 +3,54 @@ package(default_visibility = ["//visibility:public"])
 cc_library(
  

[GitHub] ajorgensen closed pull request #2864: S3Uploader improvements

[GitBox]

ajorgensen closed pull request #2864: S3Uploader improvements
URL: https://github.com/apache/incubator-heron/pull/2864
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/heron/uploaders/src/java/BUILD b/heron/uploaders/src/java/BUILD
index 109ce56d41..35074889cd 100644
--- a/heron/uploaders/src/java/BUILD
+++ b/heron/uploaders/src/java/BUILD
@@ -10,6 +10,7 @@ uploader_spi_files = [
 
 s3_deps_files = \
 uploader_spi_files + [
+"@org_apache_commons_commons_lang3//jar",
 "//third_party/java:aws-java-sdk",
 "//third_party/java:guava",
 ]
@@ -25,43 +26,43 @@ dlog_deps_files = \
 "@org_apache_distributedlog_core//jar",
 "//heron/io/dlog/src/java:dlog-lib",
 ]
-
+
 http_uploader_deps_files = \
 uploader_spi_files + [
 "@org_apache_httpcomponents_httpmime//jar",
 "@org_apache_httpcomponents_http_client//jar",
 "@org_apache_httpcomponents_http_core//jar",
 "@org_apache_commons_commons_lang3//jar",
-"//third_party/java:guava"
+"//third_party/java:guava",
 ]
 
 java_library(
-name = 'null-uploader-java',
+name = "null-uploader-java",
 srcs = glob(["**/NullUploader.java"]),
 deps = uploader_spi_files,
 )
 
 java_binary(
-name = 'null-uploader-unshaded',
+name = "null-uploader-unshaded",
 srcs = glob(["**/NullUploader.java"]),
 deps = uploader_spi_files,
 )
 
 genrule(
-name = 'heron-null-uploader',
+name = "heron-null-uploader",
 srcs = [":null-uploader-unshaded_deploy.jar"],
 outs = ["heron-null-uploader.jar"],
-cmd  = "cp $< $@",
+cmd = "cp $< $@",
 )
 
 java_library(
-name = 'localfs-uploader-java',
+name = "localfs-uploader-java",
 srcs = glob(["**/localfs/**/*.java"]),
 deps = uploader_spi_files,
 )
 
 java_binary(
-name = 'localfs-uploader-unshaded',
+name = "localfs-uploader-unshaded",
 srcs = glob(["**/localfs/**/*.java"]),
 deps = uploader_spi_files,
 )
@@ -70,17 +71,17 @@ genrule(
 name = "heron-localfs-uploader",
 srcs = [":localfs-uploader-unshaded_deploy.jar"],
 outs = ["heron-localfs-uploader.jar"],
-cmd  = "cp $< $@",
+cmd = "cp $< $@",
 )
 
 java_library(
-name = 'hdfs-uploader-java',
+name = "hdfs-uploader-java",
 srcs = glob(["**/hdfs/**/*.java"]),
 deps = uploader_spi_files,
 )
 
 java_binary(
-name = 'hdfs-uploader-unshaded',
+name = "hdfs-uploader-unshaded",
 srcs = glob(["**/hdfs/**/*.java"]),
 deps = uploader_spi_files,
 )
@@ -89,17 +90,17 @@ genrule(
 name = "heron-hdfs-uploader",
 srcs = [":hdfs-uploader-unshaded_deploy.jar"],
 outs = ["heron-hdfs-uploader.jar"],
-cmd  = "cp $< $@",
+cmd = "cp $< $@",
 )
 
 java_library(
-name = 'dlog-uploader-java',
+name = "dlog-uploader-java",
 srcs = glob(["**/dlog/**/*.java"]),
 deps = dlog_deps_files,
 )
 
 java_binary(
-name = 'dlog-uploader-unshaded',
+name = "dlog-uploader-unshaded",
 srcs = glob(["**/dlog/**/*.java"]),
 deps = dlog_deps_files,
 )
@@ -108,77 +109,81 @@ genrule(
 name = "heron-dlog-uploader",
 srcs = [":dlog-uploader-unshaded_deploy.jar"],
 outs = ["heron-dlog-uploader.jar"],
-cmd  = "cp $< $@",
+cmd = "cp $< $@",
 )
 
 java_library(
-name = 's3-uploader-java',
+name = "s3-uploader-java",
 srcs = glob(["**/s3/**/*.java"]),
 deps = s3_deps_files,
 )
 
 java_binary(
-name = 's3-uploader-unshaded',
+name = "s3-uploader-unshaded",
 srcs = glob(["**/s3/**/*.java"]),
-deps = s3_deps_files)
+deps = s3_deps_files,
+)
 
 genrule(
 name = "heron-s3-uploader",
 srcs = [":s3-uploader-unshaded_deploy.jar"],
 outs = ["heron-s3-uploader.jar"],
-cmd  = "cp $< $@",
+cmd = "cp $< $@",
 )
 
 java_library(
-name = 'scp-uploader-java',
+name = "scp-uploader-java",
 srcs = glob(["**/scp/**/*.java"]),
 deps = uploader_spi_files,
 )
 
 java_binary(
-name = 'scp-uploader-unshaded',
+name = "scp-uploader-unshaded",
 srcs = glob(["**/scp/**/*.java"]),
-deps = uploader_spi_files)
+deps = uploader_spi_files,
+)
 
 genrule(
 name = "heron-scp-uploader",
 srcs = [":scp-uploader-unshaded_deploy.jar"],
 outs = ["heron-scp-uploader.jar"],
-cmd  = "cp $< $@",
+cmd = "cp $< $@",
 )
 
 java_library(
-name = 'gcs-uploader-java',
+name = "gcs-uploader-java",
 srcs = glob(["**/gcs/**/*.java"]),
 deps = gcs_deps_files,
 )
 
 java_binary(
-name = 'gcs-uploader-unshaded',
+name = "gcs-uploader-unshaded",
 srcs = glob(["**/gcs/**/*.java"]),
-deps = gcs_deps_files)
+deps = gcs_deps_files,
+)
 
 genrule(
 name = "heron-gcs-uploader",
 srcs = [":gcs-uploader-unshaded_deploy.j

[GitHub] nlu90 closed pull request #2910: make LocalFSStorage clean checkpoints before store

[GitBox]

nlu90 closed pull request #2910: make LocalFSStorage clean checkpoints before 
store
URL: https://github.com/apache/incubator-heron/pull/2910
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/heron/config/src/yaml/conf/kubernetes/stateful.yaml 
b/heron/config/src/yaml/conf/kubernetes/stateful.yaml
index 417b882ec2..a00d9f6557 100644
--- a/heron/config/src/yaml/conf/kubernetes/stateful.yaml
+++ b/heron/config/src/yaml/conf/kubernetes/stateful.yaml
@@ -21,6 +21,7 @@ heron.statefulstorage.classname: 
"org.apache.heron.statefulstorage.localfs.Local
 heron.statefulstorage.config:
   heron.statefulstorage.classpath: ""
   heron.statefulstorage.localfs.root.path: "./checkpoints"
+  heron.statefulstorage.localfs.max.checkpoints: 10
 
 # Following are configs for socket between ckptmgr and stateful storage
 heron.ckptmgr.network.write.batch.size.bytes: 32768
diff --git a/heron/config/src/yaml/conf/local/stateful.yaml 
b/heron/config/src/yaml/conf/local/stateful.yaml
index d72e496afa..95083dd207 100644
--- a/heron/config/src/yaml/conf/local/stateful.yaml
+++ b/heron/config/src/yaml/conf/local/stateful.yaml
@@ -21,6 +21,7 @@ heron.statefulstorage.classname:
"org.apache.heron.statefulstorage.lo
 heron.statefulstorage.config:
   heron.statefulstorage.classpath:   ""
   heron.statefulstorage.localfs.root.path:   ~/.herondata/checkpoints
+  heron.statefulstorage.localfs.max.checkpoints: 10
 
 # Following are configs for socket between ckptmgr and stateful storage
 heron.ckptmgr.network.write.batch.size.bytes: 32768
diff --git a/heron/config/src/yaml/conf/localzk/stateful.yaml 
b/heron/config/src/yaml/conf/localzk/stateful.yaml
index d72e496afa..95083dd207 100644
--- a/heron/config/src/yaml/conf/localzk/stateful.yaml
+++ b/heron/config/src/yaml/conf/localzk/stateful.yaml
@@ -21,6 +21,7 @@ heron.statefulstorage.classname:
"org.apache.heron.statefulstorage.lo
 heron.statefulstorage.config:
   heron.statefulstorage.classpath:   ""
   heron.statefulstorage.localfs.root.path:   ~/.herondata/checkpoints
+  heron.statefulstorage.localfs.max.checkpoints: 10
 
 # Following are configs for socket between ckptmgr and stateful storage
 heron.ckptmgr.network.write.batch.size.bytes: 32768
diff --git a/heron/config/src/yaml/conf/marathon/stateful.yaml 
b/heron/config/src/yaml/conf/marathon/stateful.yaml
index 4ea3bfbe62..ce8785b0f3 100644
--- a/heron/config/src/yaml/conf/marathon/stateful.yaml
+++ b/heron/config/src/yaml/conf/marathon/stateful.yaml
@@ -21,6 +21,7 @@ heron.statefulstorage.classname:
"org.apache.heron.statefulstorage.lo
 heron.statefulstorage.config:
   heron.statefulstorage.classpath:   ""
   heron.statefulstorage.localfs.root.path:   "./checkpoints"
+  heron.statefulstorage.localfs.max.checkpoints: 10
 
 # Following are configs for socket between ckptmgr and stateful storage
 heron.ckptmgr.network.write.batch.size.bytes: 32768
diff --git a/heron/config/src/yaml/conf/mesos/stateful.yaml 
b/heron/config/src/yaml/conf/mesos/stateful.yaml
index 4ea3bfbe62..ce8785b0f3 100644
--- a/heron/config/src/yaml/conf/mesos/stateful.yaml
+++ b/heron/config/src/yaml/conf/mesos/stateful.yaml
@@ -21,6 +21,7 @@ heron.statefulstorage.classname:
"org.apache.heron.statefulstorage.lo
 heron.statefulstorage.config:
   heron.statefulstorage.classpath:   ""
   heron.statefulstorage.localfs.root.path:   "./checkpoints"
+  heron.statefulstorage.localfs.max.checkpoints: 10
 
 # Following are configs for socket between ckptmgr and stateful storage
 heron.ckptmgr.network.write.batch.size.bytes: 32768
diff --git a/heron/config/src/yaml/conf/nomad/stateful.yaml 
b/heron/config/src/yaml/conf/nomad/stateful.yaml
index 417b882ec2..a00d9f6557 100644
--- a/heron/config/src/yaml/conf/nomad/stateful.yaml
+++ b/heron/config/src/yaml/conf/nomad/stateful.yaml
@@ -21,6 +21,7 @@ heron.statefulstorage.classname: 
"org.apache.heron.statefulstorage.localfs.Local
 heron.statefulstorage.config:
   heron.statefulstorage.classpath: ""
   heron.statefulstorage.localfs.root.path: "./checkpoints"
+  heron.statefulstorage.localfs.max.checkpoints: 10
 
 # Following are configs for socket between ckptmgr and stateful storage
 heron.ckptmgr.network.write.batch.size.bytes: 32768
diff --git a/heron/config/src/yaml/conf/sandbox/stateful.yaml 
b/heron/config/src/yaml/conf/sandbox/stateful.yaml
index d72e496afa..95083dd207 100644
--- a/heron/config/src/yaml/conf/sandbox/stateful.yaml
+++ b/heron/config/src/yaml/conf/sandbox/stateful.yaml
@@ -21,6 +21,7 @@ heron.statefulstorage.classname:
"org.apache.heron.statefulstorage.lo
 heron.statefulstorage.config:
   heron.statefulstorage.classpath:   ""
   heron.statefulstorage.localfs.root.path:   ~/.heronda

[GitHub] kramasamy commented on issue #2850: Add SSL support for stream manager connections

[GitBox]

kramasamy commented on issue #2850: Add SSL support for stream manager 
connections
URL: https://github.com/apache/incubator-heron/pull/2850#issuecomment-394514934
 
 
   @ajorgensen - any reason for closing this?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ajorgensen commented on issue #2850: Add SSL support for stream manager connections

[GitBox]

ajorgensen commented on issue #2850: Add SSL support for stream manager 
connections
URL: https://github.com/apache/incubator-heron/pull/2850#issuecomment-394549212
 
 
   It's not complete and it has diverged enough from our implementation that I 
need to cherry-pick the commits again. I figured it'd be best to open a new PR 
with the full set of changes but I can push them here if you'd rather.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sebastienpattyn93 opened a new issue #2913: Got UploaderException when submit topology on kubernetes

[GitBox]

sebastienpattyn93 opened a new issue #2913: Got UploaderException when submit 
topology on kubernetes
URL: https://github.com/apache/incubator-heron/issues/2913
 
 
   I have a Kubernetes cluster with 3 nodes where Heron is deployed. I am able 
to visit the heron ui and to curl the heron api-server.
   
   When I try to submit a topology I keep getting the same error:
   ```
   heron submit kubernetes --verbose  ./dist/word-count-topology.pex   -   
Word_Count_Topology 
   [2018-06-07 08:08:26 +] [DEBUG]: Input Command Line Args: 
{'config_property': [], 'dry_run_format': 'colored_table', 
'topology-file-name': './dist/word-count-topology.pex', 'cluster/[role]/[env]': 
'kubernetes', 'dry_run': False, 'config_path': '/home/ubuntu/.heron/conf', 
'subcommand': 'submit', 'deploy_deactivated': False, 
'topology_main_jvm_property': [], 'extra_launch_classpath': '', 'service_url': 
'', 'topology-class-name': '-', 'verbose': 'True'}
   [2018-06-07 08:08:26 +] [DEBUG]: Input Command Line Args: 
{'config_property': [], 'dry_run_format': 'colored_table', 
'topology-file-name': './dist/word-count-topology.pex', 'cluster/[role]/[env]': 
'kubernetes', 'dry_run': False, 'config_path': '/home/ubuntu/.heron/conf', 
'subcommand': 'submit', 'deploy_deactivated': False, 
'topology_main_jvm_property': [], 'extra_launch_classpath': '', 'service_url': 
'', 'topology-class-name': '-', 'verbose': 'True'}
   [2018-06-07 08:08:26 +] [DEBUG]: Using cluster definition from file 
/home/ubuntu/.config/heron/kubernetes/cli.yaml
   [2018-06-07 08:08:26 +] [DEBUG]: Processed Command Line Args: 
{'config_property': [], 'topology-file-name': './dist/word-count-topology.pex', 
'verbose': 'True', 'subcommand': 'submit', 'deploy_deactivated': False, 
'submit_user': 'ubuntu', 'cluster': 'kubernetes', 'extra_launch_classpath': '', 
'deploy_mode': 'server', 'role': 'ubuntu', 'dry_run_format': 'colored_table', 
'dry_run': False, 'config_path': '/home/ubuntu/.heron/conf', 
'topology_main_jvm_property': [], 'environ': 'default', 'service_url': 
'http://10.128.0.20:32674', 'topology-class-name': '-'}
   [2018-06-07 08:08:26 +] [DEBUG]: Submit Args {'config_property': [], 
'topology-file-name': './dist/word-count-topology.pex', 'verbose': 'True', 
'subcommand': 'submit', 'deploy_deactivated': False, 'submit_user': 'ubuntu', 
'cluster': 'kubernetes', 'extra_launch_classpath': '', 'deploy_mode': 'server', 
'role': 'ubuntu', 'dry_run_format': 'colored_table', 'dry_run': False, 
'config_path': '/home/ubuntu/.heron/conf', 'topology_main_jvm_property': [], 
'environ': 'default', 'service_url': 'http://10.128.0.20:32674', 
'topology-class-name': '-'}
   [2018-06-07 08:08:26 +] [DEBUG]: Importing - from 
./dist/word-count-topology.pex
   [2018-06-07 08:08:26 +] [DEBUG]: Invoking class using command: 
``./dist/word-count-topology.pex Word_Count_Topology''
   [2018-06-07 08:08:26 +] [DEBUG]: Heron options: 
{cmdline.topologydefn.tmpdirectory=/tmp/tmpW58aFW,cmdline.topology.initial.state=RUNNING}
   [2018-06-07 08:08:27 +] [INFO]: Launching topology: 'Word_Count_Topology'
   [2018-06-07 08:08:27 +] [INFO]: {'config_property': [], 
'topology-file-name': './dist/word-count-topology.pex', 'verbose': 'True', 
'subcommand': 'submit', 'deploy_deactivated': False, 'submit_user': 'ubuntu', 
'cluster': 'kubernetes', 'extra_launch_classpath': '', 'deploy_mode': 'server', 
'role': 'ubuntu', 'dry_run_format': 'colored_table', 'dry_run': False, 
'config_path': '/home/ubuntu/.heron/conf', 'topology_main_jvm_property': [], 
'environ': 'default', 'service_url': 'http://10.128.0.20:32674', 
'topology-class-name': '-'}
   [2018-06-07 08:08:27 +] [DEBUG]: Starting new HTTP connection (1): 
10.128.0.20
   [2018-06-07 08:08:27 +] [DEBUG]: http://10.128.0.20:32674 "POST 
/api/v1/topologies HTTP/1.1" 500 94
   [2018-06-07 08:08:27 +] [ERROR]: Failed to instantiate uploader class 
'org.apache.heron.uploader.dlog.DLUploader'
   [2018-06-07 08:08:27 +] [ERROR]: Failed to launch topology 
'Word_Count_Topology' 
   [2018-06-07 08:08:27 +] [DEBUG]: Elapsed time: 0.409s.
   
   ```
   
   Looking in my Kubernetes POD logs I found the following error:
   
   ```
[2018-06-07 08:08:27 +] [SEVERE] 
com.twitter.heron.apiserver.resources.TopologyResource: error submitting 
topology Word_Count_Topology 
   com.twitter.heron.spi.uploader.UploaderException: Failed to instantiate 
uploader class 'org.apache.heron.uploader.dlog.DLUploader'
at 
com.twitter.heron.scheduler.SubmitterMain.submitTopology(SubmitterMain.java:411)
at 
com.twitter.heron.apiserver.actions.SubmitTopologyAction.execute(SubmitTopologyAction.java:33)
at 
com.twitter.heron.apiserver.resources.TopologyResource.submit(TopologyResource.java:223)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAcce

[GitHub] Code0x58 opened a new pull request #2914: Add permissive licence to libunwind

[GitBox]

Code0x58 opened a new pull request #2914: Add permissive licence to libunwind
URL: https://github.com/apache/incubator-heron/pull/2914
 
 
   I tried running tests with Bazel 0.14.0 and the lack of licence was a 
complaint, so I added a permissive marker as the package is from 
[LLVM](https://en.wikipedia.org/wiki/University_of_Illinois/NCSA_Open_Source_License).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] Code0x58 opened a new pull request #2915: [WIP] Use six to make heronpy tests pass in python3

[GitBox]

Code0x58 opened a new pull request #2915: [WIP] Use six to make heronpy tests 
pass in python3
URL: https://github.com/apache/incubator-heron/pull/2915
 
 
   I don't know if it is this simple, so throwing this at Travis to see if it 
sticks. I will work on getting py3 PEXs deployable in the week after next.
   
   `six` could also be used to clean up `try/except` imports in heronpy and 
possibly the other python code.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nwangtw commented on issue #2914: Add permissive licence to libunwind

[GitBox]

nwangtw commented on issue #2914: Add permissive licence to libunwind
URL: https://github.com/apache/incubator-heron/pull/2914#issuecomment-395567181
 
 
   @kramasamy 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nwangtw opened a new pull request #2916: Add DISCLAIMER file required by Apache policy

[GitBox]

nwangtw opened a new pull request #2916: Add DISCLAIMER file required by Apache 
policy
URL: https://github.com/apache/incubator-heron/pull/2916
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kramasamy closed pull request #2916: Add DISCLAIMER file required by Apache policy

[GitBox]

kramasamy closed pull request #2916: Add DISCLAIMER file required by Apache 
policy
URL: https://github.com/apache/incubator-heron/pull/2916
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/DISCLAIMER b/DISCLAIMER
new file mode 100644
index 00..0d1f308592
--- /dev/null
+++ b/DISCLAIMER
@@ -0,0 +1,6 @@
+Apache Heron (incubating) is an effort undergoing incubation at The Apache 
Software Foundation
+(ASF), sponsored by the Apache Incubator PMC. Incubation is required of all 
newly accepted
+projects until a further review indicates that the infrastructure, 
communications, and decision
+making process have stabilized in a manner consistent with other successful 
ASF projects. While
+incubation status is not necessarily a reflection of the completeness or 
stability of the code,
+it does indicate that the project has yet to be fully endorsed by the ASF.
\ No newline at end of file


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kramasamy closed pull request #2914: Add permissive licence to libunwind

[GitBox]

kramasamy closed pull request #2914: Add permissive licence to libunwind
URL: https://github.com/apache/incubator-heron/pull/2914
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/third_party/libunwind/BUILD b/third_party/libunwind/BUILD
index 9c73b707aa..9a113a32da 100644
--- a/third_party/libunwind/BUILD
+++ b/third_party/libunwind/BUILD
@@ -1,3 +1,5 @@
+licenses(["permissive"])
+
 package(default_visibility = ["//visibility:public"])
 
 exports_files([


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kramasamy commented on issue #2915: [WIP] Use six to make heronpy tests pass in python3

[GitBox]

kramasamy commented on issue #2915: [WIP] Use six to make heronpy tests pass in 
python3
URL: https://github.com/apache/incubator-heron/pull/2915#issuecomment-395777537
 
 
   Thanks for the PR. Looks like it is green across the board.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kramasamy commented on issue #2913: Got UploaderException when submit topology on kubernetes

[GitBox]

kramasamy commented on issue #2913: Got UploaderException when submit topology 
on kubernetes
URL: 
https://github.com/apache/incubator-heron/issues/2913#issuecomment-395778065
 
 
   @cckellogg - can you check this?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] cckellogg commented on issue #2913: Got UploaderException when submit topology on kubernetes

[GitBox]

cckellogg commented on issue #2913: Got UploaderException when submit topology 
on kubernetes
URL: 
https://github.com/apache/incubator-heron/issues/2913#issuecomment-395788876
 
 
   @sebastienpattyn93 what version for heron are you running? What is the yaml 
for apiserver deployment? It looks like the heron-apiserver is configured 
wrong.  I don't think there has been a release since the packing has changed. 
This class does not exist org.apache.heron.uploader.dlog.DLUploader. If you 
change the org.apache to com.twitter it should work.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] cckellogg commented on issue #2913: Got UploaderException when submit topology on kubernetes

[GitBox]

cckellogg commented on issue #2913: Got UploaderException when submit topology 
on kubernetes
URL: 
https://github.com/apache/incubator-heron/issues/2913#issuecomment-395788876
 
 
   @sebastienpattyn93 what version for heron are you running? What is the yaml 
for apiserver deployment? It looks like the heron-apiserver is configured 
wrong.  I don't think there has been a release since the packaging has changed. 
This class does not exist org.apache.heron.uploader.dlog.DLUploader. If you 
change the org.apache to com.twitter it should work.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] placeacall opened a new issue #2917: unable to setup heronpy

[GitBox]

placeacall opened a new issue #2917: unable to setup heronpy
URL: https://github.com/apache/incubator-heron/issues/2917
 
 
   
   i am having issues installing heronpy. See below
   
   @DESKTOP-9043623:~$ pip3 install heronpy
   Collecting heronpy
 Could not find a version that satisfies the requirement heronpy (from 
versions: )
   No matching distribution found for heronpy
   
   I am using Ubuntu distibution
   @DESKTOP-9043623:~$ lsb_release -a
   No LSB modules are available.
   Distributor ID: Ubuntu
   Description:Ubuntu 16.04.4 LTS
   Release:16.04
   Codename:   xenial
   
   I was able to setup heron and heron-client
   
   @DESKTOP-9043623:~$ heron version
   heron.build.git.revision : 68643ed6b880c0eb34a9704a546d5f482f517575
   heron.build.git.status : Clean
   heron.build.host : ci-server-01
   heron.build.time : Tue Feb 13 06:28:20 UTC 2018
   heron.build.timestamp : 151850330
   heron.build.user : release-agent1
   heron.build.version : 0.17.5
   
   I would appreciate advice on this, it does not seem like the usual behaviour


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sebastienpattyn93 commented on issue #2913: Got UploaderException when submit topology on kubernetes

[GitBox]

sebastienpattyn93 commented on issue #2913: Got UploaderException when submit 
topology on kubernetes
URL: 
https://github.com/apache/incubator-heron/issues/2913#issuecomment-396142312
 
 
   Hi @cckellogg I'm using the yaml from this [turotial] 
(https://apache.github.io/incubator-heron/docs/operators/deployment/schedulers/kubernetes/#general-kubernetes-clusters)
   
   ```yaml
   #  Licensed to the Apache Software Foundation (ASF) under one
   #  or more contributor license agreements.  See the NOTICE file
   #  distributed with this work for additional information
   #  regarding copyright ownership.  The ASF licenses this file
   #  to you under the Apache License, Version 2.0 (the
   #  "License"); you may not use this file except in compliance
   #  with the License.  You may obtain a copy of the License at
   #
   #http://www.apache.org/licenses/LICENSE-2.0
   #
   #  Unless required by applicable law or agreed to in writing,
   #  software distributed under the License is distributed on an
   #  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
   #  KIND, either express or implied.  See the License for the
   #  specific language governing permissions and limitations
   #  under the License.
   
   ##
   ## Heron API server deployment
   ##
   apiVersion: v1
   kind: ServiceAccount
   metadata:
 labels:
   k8s-app: heron-apiserver
 name: heron-apiserver
 namespace: default
   
   ---
   
   apiVersion: rbac.authorization.k8s.io/v1beta1
   kind: ClusterRoleBinding
   metadata:
 name: heron-apiserver
 labels:
   app: heron-apiserver
   roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: cluster-admin
   subjects:
   - kind: ServiceAccount
 name: heron-apiserver
 namespace: default
   
   ---
   
   apiVersion: extensions/v1beta1
   kind: Deployment
   metadata:
 name: heron-apiserver
 labels:
   app: heron-apiserver
 namespace: default
   spec:
 selector:
   matchLabels:
 app: heron-apiserver
 replicas: 1
 template:
   metadata:
 labels:
   app: heron-apiserver
   spec:
 serviceAccountName: heron-apiserver
 terminationGracePeriodSeconds: 0
 tolerations:
   - key: "node.kubernetes.io/not-ready"
 operator: "Equal"
 effect: "NoExecute"
 tolerationSeconds: 10
   - key: "node.alpha.kubernetes.io/notReady"
 operator: "Equal"
 effect: "NoExecute"
 tolerationSeconds: 10
   - key: "node.alpha.kubernetes.io/unreachable"
 operator: "Equal"
 effect: "NoExecute"
 tolerationSeconds: 10
 containers:
   - name: heron-apiserver
 image: heron/heron:latest
 command: ["sh", "-c"]
 args:
   - >-
 heron-apiserver
 --base-template kubernetes
 --cluster kubernetes
 -D heron.statemgr.connection.string=zookeeper:2181
 -D heron.kubernetes.scheduler.uri=http://localhost:8001
 -D heron.executor.docker.image=heron/heron:latest
 -D 
heron.class.uploader=org.apache.heron.uploader.dlog.DLUploader
 -D 
heron.uploader.dlog.topologies.namespace.uri=distributedlog://zookeeper:2181/distributedlog
   - name: kubectl-proxy
 image: heron/kubectl:latest
 command: ["sh", "-c"]
 args:
   - >
 kubectl proxy -p 8001
   ---
   apiVersion: v1
   kind: Service
   metadata:
 name: heron-apiserver
   spec:
 selector:
   app: heron-apiserver
 ports:
   - protocol: TCP
 port: 9000
 targetPort: 9000
 type: NodePort
   
   ```
   
   and this is my response when I curl the api server:
   
   ```
   {
 "heron.build.git.revision" : "ddbb98bbf173fb082c6fd575caaa35205abe34df",
 "heron.build.git.status" : "Clean",
 "heron.build.host" : "ci-server-01",
 "heron.build.time" : "Sat Mar 31 09:27:19 UTC 2018",
 "heron.build.timestamp" : "1522488439000",
 "heron.build.user" : "release-agent",
 "heron.build.version" : "0.17.8"
   }
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kramasamy commented on issue #2917: unable to setup heronpy

[GitBox]

kramasamy commented on issue #2917: unable to setup heronpy
URL: 
https://github.com/apache/incubator-heron/issues/2917#issuecomment-396250632
 
 
   @placeacall - we don't support python3 packages yet.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] jerrypeng commented on issue #2909: Add ack and fail to Streamlet API

[GitBox]

jerrypeng commented on issue #2909: Add ack and fail to Streamlet API
URL: 
https://github.com/apache/incubator-heron/issues/2909#issuecomment-396338565
 
 
   @mjschmidt so just to be clear.  The streamlet API currently does not have a 
way for users to explicitly fail a tuple but it will automatically ack tuples 
for you but there is no call back to ack at the source.  If you want, I am open 
to the idea of have a source that have ack and fail callbacks


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] jerrypeng commented on issue #2909: Add ack and fail to Streamlet API

[GitBox]

jerrypeng commented on issue #2909: Add ack and fail to Streamlet API
URL: 
https://github.com/apache/incubator-heron/issues/2909#issuecomment-396338565
 
 
   @mjschmidt so just to be clear.  The streamlet API currently does not have a 
way for users to explicitly fail a tuple but it will automatically ack tuples 
for you but there is no call back to ack at the source.  If you want, I am open 
to the idea of having a source that have ack and fail callbacks


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] placeacall closed issue #2917: unable to setup heronpy

[GitBox]

placeacall closed issue #2917: unable to setup heronpy
URL: https://github.com/apache/incubator-heron/issues/2917
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] placeacall commented on issue #2917: unable to setup heronpy

[GitBox]

placeacall commented on issue #2917: unable to setup heronpy
URL: 
https://github.com/apache/incubator-heron/issues/2917#issuecomment-396340110
 
 
   wow, works like a charm
   
   (venv) D:\Users\PycharmProjects\Heron\test>pip install heronpy
   Collecting heronpy
 Downloading 
https://files.pythonhosted.org/packages/cf/74/4707816aa9a5994b4b22c6d3707513d077728e21a5bc77a27cacfdaaf49b/heronpy-0.17.8-py2-none-any.whl
 (117kB)
   100% || 122kB 358kB/s
   Collecting protobuf==3.4.0 (from heronpy)
 Downloading 
https://files.pythonhosted.org/packages/b7/24/e7514e574b025bd86eed3bff69bf342abacd77e680a0b0c61f89a3197de3/protobuf-3.4.0-py2.py3-none-any.whl
 (375kB)
   100% || 378kB 2.6MB/s
   Collecting six>=1.9 (from protobuf==3.4.0->heronpy)
 Using cached 
https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
   Requirement already satisfied: setuptools in 
d:\users\speak\pycharmprojects\heron\test\venv\lib\site-packages (from 
protobuf==3.4.0->heronpy) (39.2.0)
   Installing collected packages: six, protobuf, heronpy
   Successfully installed heronpy-0.17.8 protobuf-3.4.0 six-1.11.0
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] placeacall commented on issue #2917: unable to setup heronpy

[GitBox]

placeacall commented on issue #2917: unable to setup heronpy
URL: 
https://github.com/apache/incubator-heron/issues/2917#issuecomment-396340110
 
 
   wow, works like a charm
   
   (venv) D:\Users\PycharmProjects\Heron\test>pip install heronpy
   Collecting heronpy
 Downloading 
https://files.pythonhosted.org/packages/cf/74/4707816aa9a5994b4b22c6d3707513d077728e21a5bc77a27cacfdaaf49b/heronpy-0.17.8-py2-none-any.whl
 (117kB)
   100% || 122kB 358kB/s
   Collecting protobuf==3.4.0 (from heronpy)
 Downloading 
https://files.pythonhosted.org/packages/b7/24/e7514e574b025bd86eed3bff69bf342abacd77e680a0b0c61f89a3197de3/protobuf-3.4.0-py2.py3-none-any.whl
 (375kB)
   100% || 378kB 2.6MB/s
   Collecting six>=1.9 (from protobuf==3.4.0->heronpy)
 Using cached 
https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
   Requirement already satisfied: setuptools in 
d:\users\speak\pycharmprojects\heron\test\venv\lib\site-packages (from 
protobuf==3.4.0->heronpy) (39.2.0)
   Installing collected packages: six, protobuf, heronpy
   Successfully installed heronpy-0.17.8 protobuf-3.4.0 six-1.11.0
   
   Thank you @kramasamy 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] mjschmidt commented on issue #2909: Add ack and fail to Streamlet API

[GitBox]

mjschmidt commented on issue #2909: Add ack and fail to Streamlet API
URL: 
https://github.com/apache/incubator-heron/issues/2909#issuecomment-396570214
 
 
   Cool. Knowing that we can get this pulled back into the master branch means 
we can work on implementing it. We will stay connected in the heronstreaming 
channel as we move forward. Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nwangtw commented on issue #2909: Add ack and fail to Streamlet API

[GitBox]

nwangtw commented on issue #2909: Add ack and fail to Streamlet API
URL: 
https://github.com/apache/incubator-heron/issues/2909#issuecomment-396696346
 
 
   And thanks for your help! :)
   
   On Tue, Jun 12, 2018 at 5:24 AM, Michael Schmidt 
   wrote:
   
   > Cool. Knowing that we can get this pulled back into the master branch
   > means we can work on implementing it. We will stay connected in the
   > heronstreaming channel as we move forward. Thanks!
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or mute the thread
   > 

   > .
   >
   
   
   
   -- 
   Regards,
   Ning Wang
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aahmed-se opened a new issue #2918: Add Vagrant based scripts to build osx binaries on linux hosts

[GitBox]

aahmed-se opened a new issue #2918: Add Vagrant based scripts to build osx 
binaries on linux hosts
URL: https://github.com/apache/incubator-heron/issues/2918
 
 
   https://www.vagrantup.com/
   
   Will probably need to target osx 10.11


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] cristobalcl opened a new issue #2919: Launch PEX on Kubernetes fails

[GitBox]

cristobalcl opened a new issue #2919: Launch PEX on Kubernetes fails
URL: https://github.com/apache/incubator-heron/issues/2919
 
 
   I recently get this error each time I try to launch a topology to Kubernetes:
   
   ```
   [2018-06-14 12:17:11 +] [DEBUG]: Starting new HTTP connection (1): 
localhost
   [2018-06-14 12:17:55 +] [DEBUG]: http://localhost:8001 "POST 
/api/v1/proxy/namespaces/heron/services/heron-apiserver:9000/api/v1/topologies 
HTTP/1.1" 500 137
   [2018-06-14 12:17:55 +] [ERROR]: Encountered exceptions on uploading the 
package 'xx.tar.gz'
   [2018-06-14 12:17:55 +] [ERROR]: Failed to launch topology 'xx' 
   [2018-06-14 12:17:55 +] [DEBUG]: Elapsed time: 44.295s.
   ```
   
   This is running locally through a kubectl proxy, but I get the same error 
when I try from another POD in the same Kubernetes cluster.
   
   ```
   $ heron version
   heron.build.git.revision : ddbb98bbf173fb082c6fd575caaa35205abe34df
   heron.build.git.status : Clean
   heron.build.host : ci-server-01
   heron.build.time : Sat Mar 31 09:25:55 UTC 2018
   heron.build.timestamp : 1522488355000
   heron.build.user : release-agent1
   heron.build.version : 0.17.8
   ```
   
   This is the log from the heron-apiserver instance in the Heron POD:
   
   
[heron-apiserver.log](https://github.com/apache/incubator-heron/files/2101959/heron-apiserver.log)
   
   I  deploy Heron in Kubernetes (AWS) using Helm:
   
   ```
   helm repo add heron-charts https://storage.googleapis.com/heron-charts
   helm install heron-charts/heron --name heron --namespace heron
   ```
   
   Of course, the topologies work OK in my local Heron.
   
   It's fail both with big (~50 mb) and small (1.5 mb) PEX. In the past, I have 
got it worked trying repeatedly 10-20 times (in a Jenkins pipeline in the same 
cluster), but now it does not work anymore.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] Code0x58 opened a new pull request #2920: Correct dag to DAG

[GitBox]

Code0x58 opened a new pull request #2920: Correct dag to DAG
URL: https://github.com/apache/incubator-heron/pull/2920
 
 
   [Do you like dags](https://www.youtube.com/watch?v=ZXILzUpVx7A)?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kramasamy commented on issue #2920: Correct dag to DAG

[GitBox]

kramasamy commented on issue #2920: Correct dag to DAG
URL: https://github.com/apache/incubator-heron/pull/2920#issuecomment-397879667
 
 
   Sure go for it!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] Code0x58 commented on a change in pull request #2920: Correct dag to DAG

[GitBox]

Code0x58 commented on a change in pull request #2920: Correct dag to DAG
URL: https://github.com/apache/incubator-heron/pull/2920#discussion_r195939946
 
 

 ##
 File path: 
heron/schedulers/src/java/org/apache/heron/scheduler/yarn/HeronMasterDriver.java
 ##
 @@ -327,7 +327,7 @@ void requestContainerForWorker(int id, final HeronWorker 
worker) {
 int cpu = worker.cores;
 ByteAmount mem = worker.mem;
 EvaluatorRequest evaluatorRequest = createEvaluatorRequest(cpu, mem);
-LOG.info(String.format("Requesting container for worker: %d, mem: %s, cpu: 
%d", id, mem, cpu));
 
 Review comment:
   Slipped in a `mem`→`RAM`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kramasamy commented on issue #2920: Correct dag to DAG

[GitBox]

kramasamy commented on issue #2920: Correct dag to DAG
URL: https://github.com/apache/incubator-heron/pull/2920#issuecomment-397932904
 
 
   @Code0x58 - is this PR ready to merge?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] Code0x58 commented on issue #2920: Correct dag to DAG

[GitBox]

Code0x58 commented on issue #2920: Correct dag to DAG
URL: https://github.com/apache/incubator-heron/pull/2920#issuecomment-397961335
 
 
   @kramasamy yep, I think it's good to go now


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] Code0x58 opened a new pull request #2921: Add JDK/JAVA_HOME to bazel_configure.py

[GitBox]

Code0x58 opened a new pull request #2921: Add JDK/JAVA_HOME to 
bazel_configure.py
URL: https://github.com/apache/incubator-heron/pull/2921
 
 
   I found that `jar` wasn't available when I did `bazel test heron/...`, so I 
thought it would be good to check for the presence of a JDK in configure.
   
   I don't know what version requirements there are, so I didn't add a check 
for that, but the version can be found from `javac -version`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] Code0x58 opened a new pull request #2922: Fix integer types to avoid warnings

[GitBox]

Code0x58 opened a new pull request #2922: Fix integer types to avoid warnings
URL: https://github.com/apache/incubator-heron/pull/2922
 
 
   The output was a bit noisy because of signed/unsigned comparisons so I 
cleaned them up.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kramasamy closed pull request #2920: Correct case of acronyms

[GitBox]

kramasamy closed pull request #2920: Correct case of acronyms
URL: https://github.com/apache/incubator-heron/pull/2920
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/.gitignore b/.gitignore
index 12fbf25f85..6091b53704 100644
--- a/.gitignore
+++ b/.gitignore
@@ -134,4 +134,7 @@ website/public/
 .vscode
 
 # integration_test
-results/
\ No newline at end of file
+results/
+
+# Vagrant
+vagrant/.vagrant/
diff --git a/WORKSPACE b/WORKSPACE
index 411f733eca..02cd7efd1c 100644
--- a/WORKSPACE
+++ b/WORKSPACE
@@ -11,7 +11,7 @@ slf4j_version = "1.7.7"
 distributedlog_version = "0.5.0"
 http_client_version = "4.5.2"
 
-# heron api server
+# heron API server
 jetty_version = "9.4.6.v20170531"
 jersey_verion = "2.25.1"
 hk2_api = "2.5.0-b32"
@@ -566,7 +566,7 @@ maven_jar(
 
 # end Kubernetes java client
 
-# heron api server
+# heron API server
 # jetty
 maven_jar(
   name = "org_eclipse_jetty_server",
@@ -723,7 +723,7 @@ maven_jar(
 )
 # end bookkeeper & distributedlog dependencies
 
-# end heron api server
+# end heron API server
 
 # Nomad dependencies
 maven_jar(
diff --git a/deploy/kubernetes/general/README.md 
b/deploy/kubernetes/general/README.md
index f6c0969d69..3b3a4d0a58 100644
--- a/deploy/kubernetes/general/README.md
+++ b/deploy/kubernetes/general/README.md
@@ -40,13 +40,13 @@ $ kubectl create -f 
https://raw.githubusercontent.com/apache/incubator-heron/mas
 $ kubectl create -f 
https://raw.githubusercontent.com/apache/incubator-heron/master/deploy/kubernetes/general/tools.yaml
 ```
 
-4. Start heron apiserver:
+4. Start heron API server:
 ```shell
 $ kubectl create -f 
https://raw.githubusercontent.com/apache/incubator-heron/master/deploy/kubernetes/general/apiserver.yaml
 ```
 
-### Deploy via heron apiserver
-We will start a proxy to the cluster and then construct a [proxy 
url](https://kubernetes.io/docs/tasks/administer-cluster/access-cluster-services/#manually-constructing-apiserver-proxy-urls)
 to access the apiserver.
+### Deploy via heron API server
+We will start a proxy to the cluster and then construct a [proxy 
url](https://kubernetes.io/docs/tasks/administer-cluster/access-cluster-services/#manually-constructing-apiserver-proxy-urls)
 to access the API server.
 
 
 1. Start kubectl proxy:
@@ -54,7 +54,7 @@ We will start a proxy to the cluster and then construct a 
[proxy url](https://ku
 $ kubectl proxy -p 8001
 ```
 
-2. Verify we can access the apiserver:
+2. Verify we can access the API server:
 
 ```shell
 $ curl 
http://localhost:8001/api/v1/proxy/namespaces/default/services/heron-apiserver:9000/api/v1/version
diff --git a/deploy/kubernetes/gke/gcs-apiserver.yaml 
b/deploy/kubernetes/gke/gcs-apiserver.yaml
index a050a49028..ee0b380833 100644
--- a/deploy/kubernetes/gke/gcs-apiserver.yaml
+++ b/deploy/kubernetes/gke/gcs-apiserver.yaml
@@ -116,7 +116,7 @@ spec:
 ---
 
 ##
-## Service to expose the heron api server
+## Service to expose the heron API server
 ##
 apiVersion: v1
 kind: Service
diff --git a/deploy/kubernetes/gke/medium.yaml 
b/deploy/kubernetes/gke/medium.yaml
index aa228f7a4b..b140d11c7f 100644
--- a/deploy/kubernetes/gke/medium.yaml
+++ b/deploy/kubernetes/gke/medium.yaml
@@ -30,7 +30,7 @@ platform: gke
 # Number of replicas for the job binary in bookkeeper
 jobReplicas: 2
 
-# amount of memory to provide for api server
+# amount of memory to provide for API server
 apiServerMemory: 512M
 
 # Number of replicas for storage bookies, memory and storage requirements 
diff --git a/deploy/kubernetes/gke/small.yaml b/deploy/kubernetes/gke/small.yaml
index f4ad4b3904..070c95f0e3 100644
--- a/deploy/kubernetes/gke/small.yaml
+++ b/deploy/kubernetes/gke/small.yaml
@@ -30,7 +30,7 @@ platform: gke
 # Number of replicas for the job binary in bookkeeper
 jobReplicas: 2
 
-# amount of memory to provide for api server
+# amount of memory to provide for API server
 apiServerMemory: 512M
 
 # Number of replicas for storage bookies, memory and storage requirements 
diff --git a/deploy/kubernetes/helm/templates/NOTES.txt 
b/deploy/kubernetes/helm/templates/NOTES.txt
index 9f89435e9d..c8bb9508eb 100644
--- a/deploy/kubernetes/helm/templates/NOTES.txt
+++ b/deploy/kubernetes/helm/templates/NOTES.txt
@@ -3,7 +3,7 @@ You can check the status of Heron by doing the following:
 1. Start kubectl proxy
kubectl proxy -p 8001
 
-2. Verify you can access the api server
+2. Verify you can access the API server
curl http://localhost:8001/api/v1/proxy/namespaces/{{ .Release.Namespace 
}}/services/{{ .Release.Name }}-apiserver:9000/api/v1/version
 
 3. Verify you can access the heron ui
diff --git a/deploy/kubernetes/helm/values.yaml.template 
b/deploy/kubernetes/helm/values.yaml.template
index 898a1e96e5..114da7da45 100644
--- a/deploy/kubernetes/helm/value

[GitHub] kramasamy commented on issue #2922: Fix integer types to avoid warnings

[GitBox]

kramasamy commented on issue #2922: Fix integer types to avoid warnings
URL: https://github.com/apache/incubator-heron/pull/2922#issuecomment-398047591
 
 
   @Code0x58 - thanks for the cleanup.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services