[jira] [Commented] (APEXCORE-405) Provide an API to launch DAG on the cluster

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359758#comment-15359758
 ] 

ASF GitHub Bot commented on APEXCORE-405:
-

Github user PramodSSImmaneni commented on a diff in the pull request:

https://github.com/apache/apex-core/pull/351#discussion_r69362973
  
--- Diff: engine/src/main/java/com/datatorrent/stram/LocalModeImpl.java ---
@@ -54,6 +55,53 @@ public DAG cloneDAG() throws Exception
   }
 
   @Override
+  public LocalAppHandle launchApp(StreamingApplication application, 
Configuration configuration, Attribute.AttributeMap
+  launchAttributes) throws LaunchException
+  {
+try {
+  //application.populateDAG(getDAG(), configuration == null ? new 
Configuration(false) : configuration);
--- End diff --

Done


> Provide an API to launch DAG on the cluster
> ---
>
> Key: APEXCORE-405
> URL: https://issues.apache.org/jira/browse/APEXCORE-405
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: Pramod Immaneni
>Assignee: Pramod Immaneni
>
> Today API exists to launch a DAG in local mode but such an API is not 
> available to launch the app on the cluster, only a CLI tool is available. 
> Provide an API to be able to do this. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] apex-core issue #351: APEXCORE-405 Common API to launch on local mode or clu...

[PramodSSImmaneni]

Github user PramodSSImmaneni commented on the issue:

https://github.com/apache/apex-core/pull/351
  
@davidyan74 please see as well


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (APEXCORE-405) Provide an API to launch DAG on the cluster

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359759#comment-15359759
 ] 

ASF GitHub Bot commented on APEXCORE-405:
-

Github user PramodSSImmaneni commented on the issue:

https://github.com/apache/apex-core/pull/351
  
@davidyan74 please see as well


> Provide an API to launch DAG on the cluster
> ---
>
> Key: APEXCORE-405
> URL: https://issues.apache.org/jira/browse/APEXCORE-405
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: Pramod Immaneni
>Assignee: Pramod Immaneni
>
> Today API exists to launch a DAG in local mode but such an API is not 
> available to launch the app on the cluster, only a CLI tool is available. 
> Provide an API to be able to do this. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] apex-core pull request #351: APEXCORE-405 Common API to launch on local mode...

[PramodSSImmaneni]

Github user PramodSSImmaneni commented on a diff in the pull request:

https://github.com/apache/apex-core/pull/351#discussion_r69362973
  
--- Diff: engine/src/main/java/com/datatorrent/stram/LocalModeImpl.java ---
@@ -54,6 +55,53 @@ public DAG cloneDAG() throws Exception
   }
 
   @Override
+  public LocalAppHandle launchApp(StreamingApplication application, 
Configuration configuration, Attribute.AttributeMap
+  launchAttributes) throws LaunchException
+  {
+try {
+  //application.populateDAG(getDAG(), configuration == null ? new 
Configuration(false) : configuration);
--- End diff --

Done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Suggestions for WindowedStream interface in the next phase

[Siyuan Hua]

Hi Community,

Right now, the WindowedStream in the pull request has following "window"
transformations:

count
countByKey
top
topByKey
combine
combineByKey
fold
foldByKey
group
groupByKey


Do you have other suggestions that are nice to have in the WindowedStream
interface.

Also I think it might be a good idea to further divide WindowedStream to
KeyedWindowedStream so developer can define some custom function to convert
normal tuple to KeyValuePair that is required by xxxByKey methods. Any
ideas on that?


Regards,
Siyuan

[jira] [Commented] (APEXMALHAR-2086) Kafka Output Operator with Kafka 0.9 API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359534#comment-15359534
 ] 

ASF GitHub Bot commented on APEXMALHAR-2086:


Github user sandeshh commented on a diff in the pull request:

https://github.com/apache/apex-malhar/pull/298#discussion_r69347651
  
--- Diff: 
kafka/src/main/java/org/apache/apex/malhar/kafka/KafkaSinglePortExactlyOnceOutputOperator.java
 ---
@@ -0,0 +1,401 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.apex.malhar.kafka;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.apex.malhar.lib.wal.FSWindowDataManager;
+import org.apache.apex.malhar.lib.wal.WindowDataManager;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.clients.producer.Callback;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
+
+import org.apache.kafka.common.PartitionInfo;
+import org.apache.kafka.common.TopicPartition;
+
+import com.datatorrent.api.Context;
+import com.datatorrent.api.DefaultInputPort;
+import com.datatorrent.api.Operator;
+
+import static 
org.apache.kafka.clients.consumer.ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG;
+import static 
org.apache.kafka.clients.consumer.ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
+import static 
org.apache.kafka.clients.consumer.ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG;
+
+/**
+ * Kafka output operator with exactly once processing semantics.
--- End diff --

@tweise & @siyuanh updated the JavaDoc. Please take a look.


> Kafka Output Operator with Kafka 0.9 API
> 
>
> Key: APEXMALHAR-2086
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2086
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>Reporter: Sandesh
>Assignee: Sandesh
>
> Goal : 2 Operartors for Kafka Output
>   1. Simple Kafka Output Operator 
> - Supports Atleast Once 
> - Expose most used producer properties as class properties
>   2. Exactly Once Kafka Output ( Not possible in all the cases, will be 
> documented later )
> 
> Design for Exactly Once
> Window Data Manager - Stores the Kafka partitions offsets.
> Kafka Key - Used by the operator = AppID#OperatorId
> During recovery. Partially written window is re-created using the following  
> approach:
> Tuples between the largest recovery offsets and the current offset are 
> checked. Based on the key, tuples written by the other entities are 
> discarded. 
> Only tuples which are not in the recovered set are emitted.
> Tuples needs to be unique within the window.
>   



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] apex-malhar pull request #298: [APEXMALHAR-2086] Kafka output operator: 0.9....

[sandeshh]

Github user sandeshh commented on a diff in the pull request:

https://github.com/apache/apex-malhar/pull/298#discussion_r69347651
  
--- Diff: 
kafka/src/main/java/org/apache/apex/malhar/kafka/KafkaSinglePortExactlyOnceOutputOperator.java
 ---
@@ -0,0 +1,401 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.apex.malhar.kafka;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.apex.malhar.lib.wal.FSWindowDataManager;
+import org.apache.apex.malhar.lib.wal.WindowDataManager;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.clients.producer.Callback;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
+
+import org.apache.kafka.common.PartitionInfo;
+import org.apache.kafka.common.TopicPartition;
+
+import com.datatorrent.api.Context;
+import com.datatorrent.api.DefaultInputPort;
+import com.datatorrent.api.Operator;
+
+import static 
org.apache.kafka.clients.consumer.ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG;
+import static 
org.apache.kafka.clients.consumer.ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
+import static 
org.apache.kafka.clients.consumer.ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG;
+
+/**
+ * Kafka output operator with exactly once processing semantics.
--- End diff --

@tweise & @siyuanh updated the JavaDoc. Please take a look.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (APEXMALHAR-2132) Rename Manifest attributes from DT-App-xxxx to Apex-App-xxxx

[Siyuan Hua (JIRA)]

Siyuan Hua created APEXMALHAR-2132:
--

 Summary: Rename Manifest attributes from DT-App- to 
Apex-App-
 Key: APEXMALHAR-2132
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2132
 Project: Apache Apex Malhar
  Issue Type: Task
Affects Versions: 4.0.0
Reporter: Siyuan Hua
 Fix For: 4.0.0


This ticket is to track the same work as 
https://issues.apache.org/jira/browse/APEXCORE-481  that needs to be done for 
malhar



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (APEXMALHAR-2131) Change the default container log name from dt.log to apex.log

[Siyuan Hua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyuan Hua closed APEXMALHAR-2131.
--
Resolution: Won't Fix

Sorry wrong project name

> Change the default container log name from dt.log to apex.log
> -
>
> Key: APEXMALHAR-2131
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2131
> Project: Apache Apex Malhar
>  Issue Type: Task
>  Components: build
>Reporter: Siyuan Hua
> Fix For: 4.0.0
>
>
> Rename dt.log to apex.loig



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (APEXMALHAR-2131) Change the default container log name from dt.log to apex.log

[Siyuan Hua (JIRA)]

Siyuan Hua created APEXMALHAR-2131:
--

 Summary: Change the default container log name from dt.log to 
apex.log
 Key: APEXMALHAR-2131
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2131
 Project: Apache Apex Malhar
  Issue Type: Task
  Components: build
Reporter: Siyuan Hua
 Fix For: 4.0.0


Rename dt.log to apex.loig



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2086) Kafka Output Operator with Kafka 0.9 API

[Siyuan Hua (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359194#comment-15359194
 ] 

Siyuan Hua commented on APEXMALHAR-2086:


[~thomas.we...@googlemail.com]
API difference
There aren't many APIs on 0.9 but still there are difference between 0.9 and 
0.8 Producer. 
In 0.9 Producer it gives you a callback saying there is some error on the 
server-side which 0.8 doesn't have. 

Reason for exactly-once operator

First of all, there is no existing exactly-once output operator in 0.8 
operators. 
Secondly, I think it is asked from some malhar users

This is actually the first exactly-once attempt for output operator with some 
assumptions (we use the keys in kafka message to skip those messages that has 
been saved but yet not recognized by WindowDataManager, this can be discussed, 
but extra space for those information is needed)

The way how it works is
When a output operator comes back from a failure, the steps are
1 It loads messages from the maximum offsets that are saved by 
WindowDataManager.
2 It starts replay messages. 
3 For messages that are within the windows saved by WindowDataManager, it 
simply skip them.
4 If the message is in last failed partial window. It use the loaded messages 
in step 1 to avoid duplication.

Because different operator partitions could write data to same kafka partition, 
we need to keep the message along with the operator partition id to recognize 
whether the messages are from the failed operator or not. We stored that 
information in the key which is a compromise we might need to think about other 
solution.
The other assumption is for step 1, we have a maximum number of messages that 
can be loaded. If the operator is not recovered very quickly and meanwhile 
other operators produce lots of data to partitions. It basically is impossible 
to dedupe the partial window mixed with data from other operators, which would 
be at least once in this case.





> Kafka Output Operator with Kafka 0.9 API
> 
>
> Key: APEXMALHAR-2086
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2086
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>Reporter: Sandesh
>Assignee: Sandesh
>
> Goal : 2 Operartors for Kafka Output
>   1. Simple Kafka Output Operator 
> - Supports Atleast Once 
> - Expose most used producer properties as class properties
>   2. Exactly Once Kafka Output ( Not possible in all the cases, will be 
> documented later )
> 
> Design for Exactly Once
> Window Data Manager - Stores the Kafka partitions offsets.
> Kafka Key - Used by the operator = AppID#OperatorId
> During recovery. Partially written window is re-created using the following  
> approach:
> Tuples between the largest recovery offsets and the current offset are 
> checked. Based on the key, tuples written by the other entities are 
> discarded. 
> Only tuples which are not in the recovered set are emitted.
> Tuples needs to be unique within the window.
>   



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2085) Implement Windowed Operators

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359095#comment-15359095
 ] 

ASF GitHub Bot commented on APEXMALHAR-2085:


Github user bhupeshchawda commented on a diff in the pull request:

https://github.com/apache/apex-malhar/pull/319#discussion_r69312802
  
--- Diff: 
library/src/main/java/org/apache/apex/malhar/lib/window/WindowedStorage.java ---
@@ -0,0 +1,88 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.apex.malhar.lib.window;
+
+import java.util.Map;
+
+import org.apache.hadoop.classification.InterfaceStability;
+
+/**
+ * WindowedStorage is a key-value store with the key being the window. The 
implementation of this interface should
+ * make sure checkpointing and recovery will be done correctly.
+ *
+ * @param  The type of the data that is stored per window
+ *
+ * TODO: Look at the possibility of integrating spillable data structure: 
https://issues.apache.org/jira/browse/APEXMALHAR-2026
--- End diff --

This seems very similar to the managed state implementations. Can we also 
look at managed state directly before looking at Spillable data structures? 


> Implement Windowed Operators
> 
>
> Key: APEXMALHAR-2085
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2085
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>Reporter: Siyuan Hua
>Assignee: David Yan
>
> As per our recent several discussions in the community. A group of Windowed 
> Operators that delivers the window semantic follows the google Data Flow 
> model(https://cloud.google.com/dataflow/) is very important. 
> The operators should be designed and implemented in a way for 
> High-level API
> Beam translation
> Easy to use with other popular operator
> {panel:title=Operator Hierarchy}
> Hierarchy of the operators,
> The windowed operators should cover all possible transformations that require 
> window, and batch processing is also considered as special window called 
> global window
> {code}
>+---+
>+-> |  WindowedOperator | <+
>|   ++--+  |
>|^  ^+
>|| | |
>|| | |
> +--+++--+--+  +---+-++--+-+
> |CombineOperator||GroupOperator|  |KeyedOperator||JoinOperator|
> +---++-+  +--+--++-+--+
>+-^   ^ ^
>| | |
>   ++---+   +-++   +++
>   |KeyedCombine|   |KeyedGroup|   | CoGroup |
>   ++   +--+   +-+
> {code}
> Combine operation includes all operations that combine all tuples in one 
> window into one or small number of tuples, Group operation group all tuples 
> in one window, Join and CoGroup are used to join and group tuples from 
> different inputs.
> {panel}
> {panel:title=Components}
> * Window Component
> It includes configuration, window state that should be checkpointed, etc. It 
> should support NonMergibleWindow(fixed or slide) MergibleWindow(Session)
> * Trigger
> It should support early trigger, late trigger with customizable trigger 
> behaviour 
> * Other related components:
> ** Watermark generator, can be plugged into input source to generate watermark
> ** Tuple schema support:
> It should handle either predefined tuple type or give a declarative API to 
> describe the user defined tuple class
> {panel}
> Most component API should be reused in High-Level API
> This is the umbrella ticket, 

[GitHub] apex-malhar pull request #319: APEXMALHAR-2085: REVIEW ONLY: Operator suppor...

[bhupeshchawda]

Github user bhupeshchawda commented on a diff in the pull request:

https://github.com/apache/apex-malhar/pull/319#discussion_r69312802
  
--- Diff: 
library/src/main/java/org/apache/apex/malhar/lib/window/WindowedStorage.java ---
@@ -0,0 +1,88 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.apex.malhar.lib.window;
+
+import java.util.Map;
+
+import org.apache.hadoop.classification.InterfaceStability;
+
+/**
+ * WindowedStorage is a key-value store with the key being the window. The 
implementation of this interface should
+ * make sure checkpointing and recovery will be done correctly.
+ *
+ * @param  The type of the data that is stored per window
+ *
+ * TODO: Look at the possibility of integrating spillable data structure: 
https://issues.apache.org/jira/browse/APEXMALHAR-2026
--- End diff --

This seems very similar to the managed state implementations. Can we also 
look at managed state directly before looking at Spillable data structures? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core issue #350: APEXCORE-222 Purging the stale data present in the buf...

[sandeshh]

Github user sandeshh commented on the issue:

https://github.com/apache/apex-core/pull/350
  
All the review comments has been addressed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (APEXMALHAR-1966) Cassandra output operator improvements

[Bhupesh Chawda (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupesh Chawda updated APEXMALHAR-1966:
---
Fix Version/s: 3.5.0

> Cassandra output operator improvements
> --
>
> Key: APEXMALHAR-1966
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-1966
> Project: Apache Apex Malhar
>  Issue Type: Improvement
>Reporter: Priyanka Gugale
>Assignee: Priyanka Gugale
> Fix For: 3.5.0
>
>
> Update existing Cassandra output operator to:
> 1. Accept use defined parameterized queries, the queries could be for update, 
> insert or delete.
> 2. Add error port to emit tuples which couldn't be written to database.
> 3. Add metrics
> 4. Provide a way to restrict batch size



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (APEXMALHAR-1966) Cassandra output operator improvements

[Bhupesh Chawda (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhupesh Chawda resolved APEXMALHAR-1966.

Resolution: Done

Merged

> Cassandra output operator improvements
> --
>
> Key: APEXMALHAR-1966
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-1966
> Project: Apache Apex Malhar
>  Issue Type: Improvement
>Reporter: Priyanka Gugale
>Assignee: Priyanka Gugale
> Fix For: 3.5.0
>
>
> Update existing Cassandra output operator to:
> 1. Accept use defined parameterized queries, the queries could be for update, 
> insert or delete.
> 2. Add error port to emit tuples which couldn't be written to database.
> 3. Add metrics
> 4. Provide a way to restrict batch size



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-1966) Cassandra output operator improvements

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358777#comment-15358777
 ] 

ASF GitHub Bot commented on APEXMALHAR-1966:


Github user asfgit closed the pull request at:

https://github.com/apache/apex-malhar/pull/295


> Cassandra output operator improvements
> --
>
> Key: APEXMALHAR-1966
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-1966
> Project: Apache Apex Malhar
>  Issue Type: Improvement
>Reporter: Priyanka Gugale
>Assignee: Priyanka Gugale
>
> Update existing Cassandra output operator to:
> 1. Accept use defined parameterized queries, the queries could be for update, 
> insert or delete.
> 2. Add error port to emit tuples which couldn't be written to database.
> 3. Add metrics
> 4. Provide a way to restrict batch size



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] apex-malhar pull request #295: APEXMALHAR-1966: Update cassandra output oper...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/apex-malhar/pull/295


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #351: APEXCORE-405 Common API to launch on local mode...

[tushargosavi]

Github user tushargosavi commented on a diff in the pull request:

https://github.com/apache/apex-core/pull/351#discussion_r69280615
  
--- Diff: engine/src/main/java/com/datatorrent/stram/LocalModeImpl.java ---
@@ -54,6 +55,53 @@ public DAG cloneDAG() throws Exception
   }
 
   @Override
+  public LocalAppHandle launchApp(StreamingApplication application, 
Configuration configuration, Attribute.AttributeMap
+  launchAttributes) throws LaunchException
+  {
+try {
+  //application.populateDAG(getDAG(), configuration == null ? new 
Configuration(false) : configuration);
--- End diff --

remove commented code.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-malhar pull request #295: APEXMALHAR-1966: Update cassandra output opre...

[DT-Priyanka]

GitHub user DT-Priyanka reopened a pull request:

https://github.com/apache/apex-malhar/pull/295

APEXMALHAR-1966: Update cassandra output opreator

1. By default populating field info from POJO
2. Option to add update/insert query

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/DT-Priyanka/incubator-apex-malhar 
APEXMALHAR-1966-cassandra-output-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/295.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #295


commit 32840a2cee5b4c9fde1c80c14df86679e47621eb
Author: Priyanka Gugale 
Date:   2016-04-01T09:20:21Z

APEXMALHAR-1966: Update casandra output opreator




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (APEXMALHAR-1953) Add generic (insert, update, delete) support to JDBC Output Operator

[Sandeep Deshmukh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-1953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sandeep Deshmukh updated APEXMALHAR-1953:
-
Fix Version/s: 3.5.0

> Add generic (insert, update, delete) support to JDBC Output Operator
> 
>
> Key: APEXMALHAR-1953
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-1953
> Project: Apache Apex Malhar
>  Issue Type: Task
>Reporter: Bhupesh Chawda
>Assignee: Bhupesh Chawda
> Fix For: 3.5.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-1953) Add generic (insert, update, delete) support to JDBC Output Operator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-1953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358610#comment-15358610
 ] 

ASF GitHub Bot commented on APEXMALHAR-1953:


Github user asfgit closed the pull request at:

https://github.com/apache/apex-malhar/pull/291


> Add generic (insert, update, delete) support to JDBC Output Operator
> 
>
> Key: APEXMALHAR-1953
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-1953
> Project: Apache Apex Malhar
>  Issue Type: Task
>Reporter: Bhupesh Chawda
>Assignee: Bhupesh Chawda
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)