[jira] [Resolved] (GOBBLIN-476) Add helix task timeout

2018-04-25 Thread Hung Tran (JIRA) 

 [ 
https://issues.apache.org/jira/browse/GOBBLIN-476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hung Tran resolved GOBBLIN-476.
---
   Resolution: Fixed
Fix Version/s: 0.13.0

Issue resolved by pull request #2347
[https://github.com/apache/incubator-gobblin/pull/2347]

> Add helix task timeout
> --
>
> Key: GOBBLIN-476
> URL: https://issues.apache.org/jira/browse/GOBBLIN-476
> Project: Apache Gobblin
>  Issue Type: Improvement
>Reporter: Kuai Yu
>Priority: Major
> Fix For: 0.13.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: GAAS feedback.

2018-04-25 Thread Vicky Kak 
Hi Abhishek,

Yes I have plans to write the wiki about it. Please try it and let know if
you get it working with the wikipedia example.

Thanks,
Vicky


On Wed, Apr 25, 2018 at 4:08 PM, Abhishek Tiwari  wrote:

> Hi Vicky,
>
> This is quite awesome work! I will try to get my hands wet with it over
> the weekend or so. Do you want to create a wiki page for this?
> Briefly looked at the commit diff and that looked fine too. I am looking
> forward to the PR.
>
> Sudarshan is working on updating his design doc to put it out for
> multi-hop.
>
> Abhishek
>
>
> On Tue, Apr 24, 2018 at 6:40 AM, Vicky Kak  wrote:
>
>> Hi Guys,
>>
>> I have created multiple JIRA's based on the discussion we had in this
>> thread, these are
>> https://issues.apache.org/jira/browse/GOBBLIN-452
>> https://issues.apache.org/jira/browse/GOBBLIN-453
>>
>> The fix for these JIRA's are done in the patch here
>> https://github.com/dallaybatta/incubator-gobblin/commit/89a1
>> 6cc1f99497525eca2889f25c60cefed06d76
>>
>> I have created the binary distribution after making these changes here
>> https://github.com/dallaybatta/uploads/releases/download/v0.
>> 13.0/gobblin-distribution-0.13.0.tar.gz
>>
>> I also encountered the following issue while docerizing the GAAS (
>> https://github.com/dallaybatta/gaas-docker)
>> https://issues.apache.org/jira/browse/GOBBLIN-472
>>
>> The dockerization of GAAS does
>> 1) Create a docker containers for GAAS+Standard Cluster(single
>> master+single worker)
>> 2) It used the Zookeeper based statestore, I started with the hadoop but
>> had issues with it.
>> 3) The docker contains contains the wikipedia.template deployed.
>> 4) It has got a script that will create the gaas container group based on
>> unique id, this unique id  ( group id) can be associated with a unique user
>> and hence provide multi tenancy.
>>
>> Here is the sample of 3 GAAS container groups being created on the same
>> machine, demonstrating the multi tenancy
>> 
>> 
>> 
>> ***
>> CONTAINER IDIMAGE COMMAND
>>   CREATED STATUS  PORTS
>>   NAMES
>> c0be6bea1451dallaybatta/gaas-cluster-worker   "/bin/sh -c
>> ./entry.s"   4 seconds ago   Up 3 seconds
>>  docker-gobblin-cluster-worker-3
>> c10644e81d5edallaybatta/gaas-cluster-master   "/bin/sh -c
>> ./entry.s"   10 seconds ago  Up 9 seconds
>>  docker-gobblin-cluster-master-3
>> 34fa6dd683c1dallaybatta/gaas-service  "/bin/sh -c
>> ./entry.s"   10 seconds ago  Up 9 seconds0.0.0.0:3->9099/tcp
>>   docker-gobblin-service-3
>> 66298f5b552eches/kafka"/start.sh"
>>   10 seconds ago  Up 9 seconds7203/tcp, 
>> 0.0.0.0:6503->9092/tcp
>>  docker-dip-kafka-3
>> 4feb9200800ajplock/zookeeper
>> "/opt/zookeeper/bin/z"   11 seconds ago  Up 10 seconds   2181/tcp,
>> 2888/tcp, 3888/tcp   docker-dip-zookeeper-3
>>
>> 65ffcbeda3f8dallaybatta/gaas-cluster-worker   "/bin/sh -c
>> ./entry.s"   16 seconds ago  Up 16 seconds
>> docker-gobblin-cluster-worker-2
>> 8702911890dbdallaybatta/gaas-cluster-master   "/bin/sh -c
>> ./entry.s"   22 seconds ago  Up 21 seconds
>> docker-gobblin-cluster-master-2
>> 74e2f2040c68dallaybatta/gaas-service  "/bin/sh -c
>> ./entry.s"   22 seconds ago  Up 21 seconds   0.0.0.0:2->9099/tcp
>>   docker-gobblin-service-2
>> 55ef0b4e7366ches/kafka"/start.sh"
>>   22 seconds ago  Up 22 seconds   7203/tcp, 
>> 0.0.0.0:6502->9092/tcp
>>  docker-dip-kafka-2
>> 123ce726fc5ajplock/zookeeper
>> "/opt/zookeeper/bin/z"   23 seconds ago  Up 22 seconds   2181/tcp,
>> 2888/tcp, 3888/tcp   docker-dip-zookeeper-2
>>
>> 1af34cfb5cb0dallaybatta/gaas-cluster-worker   "/bin/sh -c
>> ./entry.s"   31 seconds ago  Up 30 seconds
>> docker-gobblin-cluster-worker-1
>> bcf010f19338dallaybatta/gaas-cluster-master   "/bin/sh -c
>> ./entry.s"   36 seconds ago  Up 35 seconds
>> docker-gobblin-cluster-master-1
>> 1e10303e012adallaybatta/gaas-service  "/bin/sh -c
>> ./entry.s"   36 seconds ago  Up 36 seconds   0.0.0.0:1->9099/tcp
>>   docker-gobblin-service-1
>> b2a4f6ce6f86ches/kafka"/start.sh"
>>   37 seconds ago  Up 36 seconds   7203/tcp, 
>> 0.0.0.0:6501->9092/tcp
>>  docker-dip-kafka-1
>> f6b92760c3a7jplock/zookeeper
>> "/opt/zookeeper/bin/z"   37 seconds ago  Up 36 seconds   2181/tcp,
>> 2888/tcp, 3888/tcp   docker-dip-zookeeper-1
>>
>>

Re: GAAS feedback.

2018-04-25 Thread Abhishek Tiwari 
Hi Vicky,

This is quite awesome work! I will try to get my hands wet with it over the
weekend or so. Do you want to create a wiki page for this?
Briefly looked at the commit diff and that looked fine too. I am looking
forward to the PR.

Sudarshan is working on updating his design doc to put it out for
multi-hop.

Abhishek

On Tue, Apr 24, 2018 at 6:40 AM, Vicky Kak  wrote:

> Hi Guys,
>
> I have created multiple JIRA's based on the discussion we had in this
> thread, these are
> https://issues.apache.org/jira/browse/GOBBLIN-452
> https://issues.apache.org/jira/browse/GOBBLIN-453
>
> The fix for these JIRA's are done in the patch here
> https://github.com/dallaybatta/incubator-gobblin/commit/89a1
> 6cc1f99497525eca2889f25c60cefed06d76
>
> I have created the binary distribution after making these changes here
> https://github.com/dallaybatta/uploads/releases/download/v0.
> 13.0/gobblin-distribution-0.13.0.tar.gz
>
> I also encountered the following issue while docerizing the GAAS (
> https://github.com/dallaybatta/gaas-docker)
> https://issues.apache.org/jira/browse/GOBBLIN-472
>
> The dockerization of GAAS does
> 1) Create a docker containers for GAAS+Standard Cluster(single
> master+single worker)
> 2) It used the Zookeeper based statestore, I started with the hadoop but
> had issues with it.
> 3) The docker contains contains the wikipedia.template deployed.
> 4) It has got a script that will create the gaas container group based on
> unique id, this unique id  ( group id) can be associated with a unique user
> and hence provide multi tenancy.
>
> Here is the sample of 3 GAAS container groups being created on the same
> machine, demonstrating the multi tenancy
> 
> 
> 
> ***
> CONTAINER IDIMAGE COMMAND
> CREATED STATUS  PORTS
> NAMES
> c0be6bea1451dallaybatta/gaas-cluster-worker   "/bin/sh -c
> ./entry.s"   4 seconds ago   Up 3 seconds
>  docker-gobblin-cluster-worker-3
> c10644e81d5edallaybatta/gaas-cluster-master   "/bin/sh -c
> ./entry.s"   10 seconds ago  Up 9 seconds
>  docker-gobblin-cluster-master-3
> 34fa6dd683c1dallaybatta/gaas-service  "/bin/sh -c
> ./entry.s"   10 seconds ago  Up 9 seconds0.0.0.0:3->9099/tcp
>   docker-gobblin-service-3
> 66298f5b552eches/kafka"/start.sh"
> 10 seconds ago  Up 9 seconds7203/tcp, 0.0.0.0:6503->9092/tcp
>  docker-dip-kafka-3
> 4feb9200800ajplock/zookeeper
> "/opt/zookeeper/bin/z"   11 seconds ago  Up 10 seconds   2181/tcp,
> 2888/tcp, 3888/tcp   docker-dip-zookeeper-3
>
> 65ffcbeda3f8dallaybatta/gaas-cluster-worker   "/bin/sh -c
> ./entry.s"   16 seconds ago  Up 16 seconds
> docker-gobblin-cluster-worker-2
> 8702911890dbdallaybatta/gaas-cluster-master   "/bin/sh -c
> ./entry.s"   22 seconds ago  Up 21 seconds
> docker-gobblin-cluster-master-2
> 74e2f2040c68dallaybatta/gaas-service  "/bin/sh -c
> ./entry.s"   22 seconds ago  Up 21 seconds   0.0.0.0:2->9099/tcp
>   docker-gobblin-service-2
> 55ef0b4e7366ches/kafka"/start.sh"
> 22 seconds ago  Up 22 seconds   7203/tcp, 0.0.0.0:6502->9092/tcp
>  docker-dip-kafka-2
> 123ce726fc5ajplock/zookeeper
> "/opt/zookeeper/bin/z"   23 seconds ago  Up 22 seconds   2181/tcp,
> 2888/tcp, 3888/tcp   docker-dip-zookeeper-2
>
> 1af34cfb5cb0dallaybatta/gaas-cluster-worker   "/bin/sh -c
> ./entry.s"   31 seconds ago  Up 30 seconds
> docker-gobblin-cluster-worker-1
> bcf010f19338dallaybatta/gaas-cluster-master   "/bin/sh -c
> ./entry.s"   36 seconds ago  Up 35 seconds
> docker-gobblin-cluster-master-1
> 1e10303e012adallaybatta/gaas-service  "/bin/sh -c
> ./entry.s"   36 seconds ago  Up 36 seconds   0.0.0.0:1->9099/tcp
>   docker-gobblin-service-1
> b2a4f6ce6f86ches/kafka"/start.sh"
> 37 seconds ago  Up 36 seconds   7203/tcp, 0.0.0.0:6501->9092/tcp
>  docker-dip-kafka-1
> f6b92760c3a7jplock/zookeeper
> "/opt/zookeeper/bin/z"   37 seconds ago  Up 36 seconds   2181/tcp,
> 2888/tcp, 3888/tcp   docker-dip-zookeeper-1
>
> 
> 
> 
> ***
>
> Please note that it just uses the GAAS's rest endpoint and the Standalone
> Cluster for scalibility
>
> Also we could deploy the artifacts ( tempaltes into the gaas-service
> container and

[jira] [Created] (GOBBLIN-476) Add helix task timeout

2018-04-25 Thread Kuai Yu (JIRA) 
Kuai Yu created GOBBLIN-476:
---

 Summary: Add helix task timeout
 Key: GOBBLIN-476
 URL: https://issues.apache.org/jira/browse/GOBBLIN-476
 Project: Apache Gobblin
  Issue Type: Improvement
Reporter: Kuai Yu






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)