Re: Using Docker images to run Thrift

[Apoorv Palkar]

+1, IDK what thrift and/or docker are(gourav explained me), shoutout marcus for 
being smart.



-Original Message-
From: Christie, Marcus Aaron 
To: dev 
Sent: Wed, Jun 7, 2017 3:02 pm
Subject: Using Docker images to run Thrift


Dev,


After running into difficulties getting Thrift to build on my laptop I started 
exploring the possibility of using Docker images to run Thrift.  I’ve created a 
pull request of my changes here: https://github.com/apache/airavata/pull/112


One question: I opted to just switch the scripts to using Docker, but I thought 
perhaps that could be a command line flag whether to use Docker or not.  My 
hope is that using Docker images to run Thrift will be a lot more convenient 
than requiring developers to install Thrift.


Your feedback is welcome.


Thanks,


Marcus

Re: Upgrade to Thrift 0.10

[Suresh Marru]

Thanks Anuj for raising these. Its important to have backward compatibility 
with older clients I will first try them before committing my changes.

Suresh

> On Jun 7, 2017, at 2:33 PM, Anuj Bhandar  wrote:
> 
> I tried working with Thrift 0.10, but was facing difficulties with backward 
> compatibility with stubs generated from Thrift 0.93 and vice-versa. We have 
> to make sure this is not a problem before we make a decision.
> 
> Thanks and best regards,
> Anuj Bhandar
> 
> On 6/7/17 7:41 AM, Suresh Marru wrote:
>> Hi All,
>> 
>> Thrift 0.10 has been stable. I will change the develop branch to do this 
>> upgrade. Please suggest if you see any concerns with this upgrade. The key 
>> motivation is to leverage Python 3 support in thrift 0.10.
>> 
>> Suresh
> 

Re: Using Docker images to run Thrift

[Suresh Marru]

+ 1 for docker support and also for having an option to use native thrift 
install.

Suresh

> On Jun 7, 2017, at 3:24 PM, Shenoy, Gourav Ganesh  
> wrote:
> 
> +1
>  
> Thanks and Regards,
> Gourav Shenoy
>  
> From: "Pamidighantam, Sudhakar V" 
> Reply-To: "dev@airavata.apache.org" 
> Date: Wednesday, June 7, 2017 at 3:15 PM
> To: "dev@airavata.apache.org" 
> Subject: Re: Using Docker images to run Thrift
>  
> +1 any thing to reduce manual installation and corresponding issues would be 
> welcome. There should be testing to ensure this went through correctly and 
> suggest practical ways to correct any shortcomings. 
>  
> Thanks,
> Sudhakar.
> On Jun 7, 2017, at 3:02 PM, Christie, Marcus Aaron  > wrote:
>  
> Dev, 
>  
> After running into difficulties getting Thrift to build on my laptop I 
> started exploring the possibility of using Docker images to run Thrift.  I’ve 
> created a pull request of my changes here: 
> https://github.com/apache/airavata/pull/112 
> 
>  
> One question: I opted to just switch the scripts to using Docker, but I 
> thought perhaps that could be a command line flag whether to use Docker or 
> not.  My hope is that using Docker images to run Thrift will be a lot more 
> convenient than requiring developers to install Thrift.
>  
> Your feedback is welcome.
>  
> Thanks,
>  
> Marcus
>  

Re: Using Docker images to run Thrift

[Shenoy, Gourav Ganesh]

+1

Thanks and Regards,
Gourav Shenoy

From: "Pamidighantam, Sudhakar V" 
Reply-To: "dev@airavata.apache.org" 
Date: Wednesday, June 7, 2017 at 3:15 PM
To: "dev@airavata.apache.org" 
Subject: Re: Using Docker images to run Thrift

+1 any thing to reduce manual installation and corresponding issues would be 
welcome. There should be testing to ensure this went through correctly and 
suggest practical ways to correct any shortcomings.

Thanks,
Sudhakar.
On Jun 7, 2017, at 3:02 PM, Christie, Marcus Aaron 
mailto:machr...@iu.edu>> wrote:

Dev,

After running into difficulties getting Thrift to build on my laptop I started 
exploring the possibility of using Docker images to run Thrift.  I’ve created a 
pull request of my changes here: 
https://github.com/apache/airavata/pull/112

One question: I opted to just switch the scripts to using Docker, but I thought 
perhaps that could be a command line flag whether to use Docker or not.  My 
hope is that using Docker images to run Thrift will be a lot more convenient 
than requiring developers to install Thrift.

Your feedback is welcome.

Thanks,

Marcus

Re: Using Docker images to run Thrift

[Pamidighantam, Sudhakar V]

+1 any thing to reduce manual installation and corresponding issues would be 
welcome. There should be testing to ensure this went through correctly and 
suggest practical ways to correct any shortcomings.

Thanks,
Sudhakar.
On Jun 7, 2017, at 3:02 PM, Christie, Marcus Aaron 
mailto:machr...@iu.edu>> wrote:

Dev,

After running into difficulties getting Thrift to build on my laptop I started 
exploring the possibility of using Docker images to run Thrift.  I’ve created a 
pull request of my changes here: 
https://github.com/apache/airavata/pull/112

One question: I opted to just switch the scripts to using Docker, but I thought 
perhaps that could be a command line flag whether to use Docker or not.  My 
hope is that using Docker images to run Thrift will be a lot more convenient 
than requiring developers to install Thrift.

Your feedback is welcome.

Thanks,

Marcus

Re: Using Docker images to run Thrift

[Supun Nakandala]

+1

On Wed, Jun 7, 2017 at 3:02 PM, Christie, Marcus Aaron 
wrote:

> Dev,
>
> After running into difficulties getting Thrift to build on my laptop I
> started exploring the possibility of using Docker images to run Thrift.
> I’ve created a pull request of my changes here: https://github.com/
> apache/airavata/pull/112
>
> One question: I opted to just switch the scripts to using Docker, but I
> thought perhaps that could be a command line flag whether to use Docker or
> not.  My hope is that using Docker images to run Thrift will be a lot more
> convenient than requiring developers to install Thrift.
>
> Your feedback is welcome.
>
> Thanks,
>
> Marcus
>



-- 
Thank you
Supun Nakandala
Dept. Computer Science and Engineering
University of Moratuwa

Using Docker images to run Thrift

[Christie, Marcus Aaron]

Dev,

After running into difficulties getting Thrift to build on my laptop I started 
exploring the possibility of using Docker images to run Thrift.  I’ve created a 
pull request of my changes here: https://github.com/apache/airavata/pull/112

One question: I opted to just switch the scripts to using Docker, but I thought 
perhaps that could be a command line flag whether to use Docker or not.  My 
hope is that using Docker images to run Thrift will be a lot more convenient 
than requiring developers to install Thrift.

Your feedback is welcome.

Thanks,

Marcus

Re: Upgrade to Thrift 0.10

[Anuj Bhandar]

I tried working with Thrift 0.10, but was facing difficulties with 
backward compatibility with stubs generated from Thrift 0.93 and 
vice-versa. We have to make sure this is not a problem before we make a 
decision.


Thanks and best regards,
Anuj Bhandar

On 6/7/17 7:41 AM, Suresh Marru wrote:

Hi All,

Thrift 0.10 has been stable. I will change the develop branch to do this 
upgrade. Please suggest if you see any concerns with this upgrade. The key 
motivation is to leverage Python 3 support in thrift 0.10.

Suresh

Re: Apache Flink Execution

[Shenoy, Gourav Ganesh]

Hi dev,

I did some literature reading about Storm vs Flink, with an emphasis on our 
use-case of Distributed Task Execution and my initial impressions are as 
follows (I will also be updating the Google docs accordingly):


1.   Although both Storm and Flink engines appear to be similar, for 
supporting pipeline processing; Storm can only handle data streams, whereas 
Flink supports stream and batch processing. This allows Flink to perform data 
transfer between parallel tasks – we do not have such support as of today, but 
we can definitely think of parallel task execution.

2.   Storm supports at-least once and at-most once data processing, whereas 
Flink guarantees exactly-once processing. Storm also supports exactly-once via 
their Trident API. From what I read, Flink claims to be more efficient in terms 
of processing semantics – as they use a lighter algorithm for check-pointing 
data transfers.

3.   There are high level APIs available in Flink to simplify the data 
collection process, which is a little tedious in Storm. In Storm one needs to 
manually implement readers and collectors, whereas Flink provides functions 
such as Map, GroupBy, Window and Join.

4.   A major positive in Flink is the ability to maintain custom State 
information in operators/executors. This custom state information can also be 
used in check-pointing for fault tolerance.

I think Flink is an improvement over Storm, but this is just an understanding 
from my initial readings. I haven’t yet tried coding any examples in Flink. 
Again, most of the features/differences mentioned above, offered by both Storm 
and Flink, are for stream processing with focus on executing a large number of 
small tasks (in parallel?) with continuous streaming data and therefore the 
fight is for offering low latency processing; these might not necessarily be 
that important for the Airavata use-case (tasks may take time to complete).

Thanks and Regards,
Gourav Shenoy

From: "Pierce, Marlon" 
Reply-To: 
Date: Wednesday, May 24, 2017 at 11:36 AM
To: "dev@airavata.apache.org" 
Subject: Re: Apache Flink Execution

Thanks, Apoorv.  Note for everyone else: request access if you’d like to leave 
a comment or make a suggestion.

Marlon

From: Apoorv Palkar 
Reply-To: "dev@airavata.apache.org" 
Date: Wednesday, May 24, 2017 at 11:32 AM
To: "dev@airavata.apache.org" 
Subject: Apache Flink Execution

https://docs.google.com/document/d/1GDh8kEbAXVY9Gv1mmFvq__zLN_JP6m2_KbfN-9C0uO0/edit?usp=sharing

LINK for Flink Use/fundamental

Upgrade to Thrift 0.10

[Suresh Marru]

Hi All,

Thrift 0.10 has been stable. I will change the develop branch to do this 
upgrade. Please suggest if you see any concerns with this upgrade. The key 
motivation is to leverage Python 3 support in thrift 0.10.

Suresh