[jira] [Commented] (MYRIAD-275) Upgrade front dependencies
[ https://issues.apache.org/jira/browse/MYRIAD-275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688119#comment-16688119 ] Juan Pedro commented on MYRIAD-275: --- I think it is necessary to update the dependencies with which front works (package.json). Versions of gulp, node, etc ... We are working with old versions. I think it would also be interesting to update / refactor / redesign the front to make it more attractive ... What do you think? > Upgrade front dependencies > -- > > Key: MYRIAD-275 > URL: https://issues.apache.org/jira/browse/MYRIAD-275 > Project: Myriad > Issue Type: Task > Components: Scheduler >Affects Versions: Myriad 0.3.0 >Reporter: Juan Pedro >Priority: Major > Labels: build > Fix For: Myriad 0.3.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MYRIAD-276) Test Myriad with Hadoop 3
[ https://issues.apache.org/jira/browse/MYRIAD-276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juan Pedro updated MYRIAD-276: -- Issue Type: Task (was: Wish) > Test Myriad with Hadoop 3 > - > > Key: MYRIAD-276 > URL: https://issues.apache.org/jira/browse/MYRIAD-276 > Project: Myriad > Issue Type: Task >Reporter: Juan Pedro >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MYRIAD-276) Test Myriad with Hadoop 3
Juan Pedro created MYRIAD-276: - Summary: Test Myriad with Hadoop 3 Key: MYRIAD-276 URL: https://issues.apache.org/jira/browse/MYRIAD-276 Project: Myriad Issue Type: Wish Reporter: Juan Pedro -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MYRIAD-275) Upgrade front dependencies
Juan Pedro created MYRIAD-275: - Summary: Upgrade front dependencies Key: MYRIAD-275 URL: https://issues.apache.org/jira/browse/MYRIAD-275 Project: Myriad Issue Type: Task Components: Scheduler Reporter: Juan Pedro -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MYRIAD-273) upgrade gradle version
Juan Pedro created MYRIAD-273: - Summary: upgrade gradle version Key: MYRIAD-273 URL: https://issues.apache.org/jira/browse/MYRIAD-273 Project: Myriad Issue Type: Wish Reporter: Juan Pedro -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MYRIAD-270) Develop an EventHandler to perform the StatusUpdate ack
[ https://issues.apache.org/jira/browse/MYRIAD-270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juan Pedro updated MYRIAD-270: -- Summary: Develop an EventHandler to perform the StatusUpdate ack (was: Develop an EventHabdler to perform the StatusUpdate ack) > Develop an EventHandler to perform the StatusUpdate ack > --- > > Key: MYRIAD-270 > URL: https://issues.apache.org/jira/browse/MYRIAD-270 > Project: Myriad > Issue Type: New Feature > Components: Scheduler >Affects Versions: Myriad 0.3.0 > Environment: upgrademesos1.5 branch >Reporter: Juan Pedro >Priority: Minor > > With the update of the version of mesos it is necessary to realize an ack > when a status update is received -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MYRIAD-272) migrate repo to gitbox
Juan Pedro created MYRIAD-272: - Summary: migrate repo to gitbox Key: MYRIAD-272 URL: https://issues.apache.org/jira/browse/MYRIAD-272 Project: Myriad Issue Type: Improvement Reporter: Juan Pedro Currently repos are beginning to migrate to gitbox. I think it would be very interesting to take advantage of the advantages -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MYRIAD-271) supress-revive mechanism offers
[ https://issues.apache.org/jira/browse/MYRIAD-271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juan Pedro updated MYRIAD-271: -- Environment: upgrademesos1.5 branch > supress-revive mechanism offers > --- > > Key: MYRIAD-271 > URL: https://issues.apache.org/jira/browse/MYRIAD-271 > Project: Myriad > Issue Type: New Feature > Components: Scheduler >Affects Versions: Myriad 0.3.0 > Environment: upgrademesos1.5 branch >Reporter: Juan Pedro >Priority: Minor > > introduce a mechanism to suppress offers when they are not necessary and > revive them when there are tasks to deploy -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MYRIAD-270) Develop an EventHabdler to perform the StatusUpdate ack
[ https://issues.apache.org/jira/browse/MYRIAD-270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juan Pedro updated MYRIAD-270: -- Environment: upgrademesos1.5 branch > Develop an EventHabdler to perform the StatusUpdate ack > --- > > Key: MYRIAD-270 > URL: https://issues.apache.org/jira/browse/MYRIAD-270 > Project: Myriad > Issue Type: New Feature > Components: Scheduler >Affects Versions: Myriad 0.3.0 > Environment: upgrademesos1.5 branch >Reporter: Juan Pedro >Priority: Minor > > With the update of the version of mesos it is necessary to realize an ack > when a status update is received -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MYRIAD-270) Develop an EventHabdler to perform the StatusUpdate ack
[ https://issues.apache.org/jira/browse/MYRIAD-270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juan Pedro updated MYRIAD-270: -- Affects Version/s: Myriad 0.3.0 > Develop an EventHabdler to perform the StatusUpdate ack > --- > > Key: MYRIAD-270 > URL: https://issues.apache.org/jira/browse/MYRIAD-270 > Project: Myriad > Issue Type: New Feature > Components: Scheduler >Affects Versions: Myriad 0.3.0 >Reporter: Juan Pedro >Priority: Minor > > With the update of the version of mesos it is necessary to realize an ack > when a status update is received -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MYRIAD-271) supress-revive mechanism offers
Juan Pedro created MYRIAD-271: - Summary: supress-revive mechanism offers Key: MYRIAD-271 URL: https://issues.apache.org/jira/browse/MYRIAD-271 Project: Myriad Issue Type: New Feature Components: Scheduler Affects Versions: Myriad 0.3.0 Reporter: Juan Pedro introduce a mechanism to suppress offers when they are not necessary and revive them when there are tasks to deploy -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MYRIAD-270) Develop an EventHabdler to perform the StatusUpdate ack
Juan Pedro created MYRIAD-270: - Summary: Develop an EventHabdler to perform the StatusUpdate ack Key: MYRIAD-270 URL: https://issues.apache.org/jira/browse/MYRIAD-270 Project: Myriad Issue Type: New Feature Components: Scheduler Reporter: Juan Pedro With the update of the version of mesos it is necessary to realize an ack when a status update is received -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MYRIAD-269) Generate json deployment for DCOS universe
Juan Pedro created MYRIAD-269: - Summary: Generate json deployment for DCOS universe Key: MYRIAD-269 URL: https://issues.apache.org/jira/browse/MYRIAD-269 Project: Myriad Issue Type: Improvement Reporter: Juan Pedro Generate json deployment for DCOS universe -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MYRIAD-264) Upgrade Mesos API to 1.5.x
[ https://issues.apache.org/jira/browse/MYRIAD-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610461#comment-16610461 ] Juan Pedro commented on MYRIAD-264: --- Hi [~yufeldman], thanks for the help, I had not thought about shading and it could be a good option > Upgrade Mesos API to 1.5.x > -- > > Key: MYRIAD-264 > URL: https://issues.apache.org/jira/browse/MYRIAD-264 > Project: Myriad > Issue Type: Task > Components: Executor, Scheduler >Affects Versions: Myriad 0.2.0 >Reporter: Javi Roman >Assignee: Javi Roman >Priority: Critical > > Apache Myriad is based on Apache Mesos API 0.28.x [Upgrading > Mesos|https://mesos.apache.org/documentation/latest/upgrades/]. > In order to align with current Apache Mesos deployments we have to upgrade to > "at least" Apache Mesos 1.5.x API (this is the Mesos version shipped in DC/OS > 1.11.1, in particular 1.5.1-dev) [DC/OS 1.11.1 > Mesos|https://github.com/dcos/dcos/blob/1.11.1/packages/mesos/buildinfo.json] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MYRIAD-264) Upgrade Mesos API to 1.5.x
[ https://issues.apache.org/jira/browse/MYRIAD-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605945#comment-16605945 ] Juan Pedro commented on MYRIAD-264: --- Hello, I am working on version alignment with DCOS. The version of protobuf that Yarn uses and the one that uses Mesos are not compatible. The jar of Mesos is compiled for 3.x and Yarn charges 2.5. This causes Myriad to find the protobuf classes already loaded in a version that is not correct for Mesos. In this context, I have developed two lines of work: 1. I have developed a proprietary driver to manage the connection with Mesos so that we can compile with the appropriate protobuf version. This supposes freedom but also supposes an additional maintenance (abandoned in favor of the second option). https://github.com/jpgilaberte/incubator-myriad/tree/httpapi (compile without test - only developed the scheduler driver) 2. Reuse all the code of the original mesos driver, compile our protobuf objects and continue to take advantage of the binding with libmesos.so. The current state of this branch is that it is already running with Mesos 1.5. I continue to work on adding the new features that have been integrated into the latest versions of Mesos (StatusUpdateAcknoweledge, etc...). https://github.com/jpgilaberte/incubator-myriad/tree/upgrademesos1.5 What do you think? > Upgrade Mesos API to 1.5.x > -- > > Key: MYRIAD-264 > URL: https://issues.apache.org/jira/browse/MYRIAD-264 > Project: Myriad > Issue Type: Task > Components: Executor, Scheduler >Affects Versions: Myriad 0.2.0 >Reporter: Javi Roman >Assignee: Javi Roman >Priority: Critical > > Apache Myriad is based on Apache Mesos API 0.28.x [Upgrading > Mesos|https://mesos.apache.org/documentation/latest/upgrades/]. > In order to align with current Apache Mesos deployments we have to upgrade to > "at least" Apache Mesos 1.5.x API (this is the Mesos version shipped in DC/OS > 1.11.1, in particular 1.5.1-dev) [DC/OS 1.11.1 > Mesos|https://github.com/dcos/dcos/blob/1.11.1/packages/mesos/buildinfo.json] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MYRIAD-265) Exponential backoff reconcilation service
Juan Pedro created MYRIAD-265: - Summary: Exponential backoff reconcilation service Key: MYRIAD-265 URL: https://issues.apache.org/jira/browse/MYRIAD-265 Project: Myriad Issue Type: Improvement Components: Scheduler Reporter: Juan Pedro //TODO(mohit): Using exponential backoff here (ReconcileService:66), maybe backoff strategy should be configurable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MYRIAD-88) Reduce verbose logging
[ https://issues.apache.org/jira/browse/MYRIAD-88?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437205#comment-16437205 ] Juan Pedro commented on MYRIAD-88: -- This issue is already corrected, the level of the log in the class mentioned to debug has been changed. I have made an analysis of the log and for me it is correct. The INFO level, informs what Myriad is doing. I propose to change the following logs from INFO to WARN: - NMExecutorCommandLineGenerator: 146 - DisconnectedEventHandler: 34 - ExecutorLostEventHandler: 39 - SlaveLostEventHandler: 37 I think they are warnings of possible errors due to misconfiguration and warnings of 'anomalous operation' in third-party systems. In this way the WARN level is more concrete when there is something 'weird'. Also from WARN to ERROR: - Main: 294 > Reduce verbose logging > -- > > Key: MYRIAD-88 > URL: https://issues.apache.org/jira/browse/MYRIAD-88 > Project: Myriad > Issue Type: Improvement >Reporter: Santosh Marella > > One instance of "INFO" logging that should perhaps be at "DEBUG" level is > 15/04/06 08:15:52 INFO handlers.ResourceOffersEventHandler: Received offers 2 > 15/04/06 08:15:52 INFO handlers.ResourceOffersEventHandler: No pending > tasks, declining all offers > 15/04/06 08:15:55 INFO handlers.ResourceOffersEventHandler: Received offers 2 > 15/04/06 08:15:55 INFO handlers.ResourceOffersEventHandler: No pending > tasks, declining all offers > 15/04/06 08:15:56 INFO handlers.ResourceOffersEventHandler: Received offers 1 > 15/04/06 08:15:56 INFO handlers.ResourceOffersEventHandler: No pending > tasks, declining all offers > There might be other instances where Myriad is logging too much. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MYRIAD-12) Implement auto scaling
[ https://issues.apache.org/jira/browse/MYRIAD-12?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437126#comment-16437126 ] Juan Pedro commented on MYRIAD-12: -- Hi. Other ideas: - Option to limit the scaling according to the time (window for batch jobs). - Scale by zones, according to constraints, where long-lived processes may not need resources (night time). - Scale on certain nodes, scaling nodes (according to constraints), where the need for resources varies depending on the time or workload. > Implement auto scaling > -- > > Key: MYRIAD-12 > URL: https://issues.apache.org/jira/browse/MYRIAD-12 > Project: Myriad > Issue Type: Improvement >Reporter: Santosh Marella >Assignee: Santosh Marella > > - Pluggable criteria to detect "YARN starvation".. some ideas: > - if all NMs are full and more than X number of resource requests waiting > in the queue > - if all NMs are full and more than Y number of apps are waiting in the > queue > - How to scale? i.e. "how many" NM instances are required to bring down the > starvation > - "detect the extent of starvation" by querying the resource requests in > the pipieline and scale up accordingly -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MYRIAD-39) Constraint-based scheduling
[ https://issues.apache.org/jira/browse/MYRIAD-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437087#comment-16437087 ] Juan Pedro commented on MYRIAD-39: -- Hi. I think it would be very interesting to work on the constraints of Marathon and understand what options are more interesting for Myriad. http://mesosphere.github.io/marathon/docs/constraints.html > Constraint-based scheduling > --- > > Key: MYRIAD-39 > URL: https://issues.apache.org/jira/browse/MYRIAD-39 > Project: Myriad > Issue Type: Improvement >Reporter: Adam B > > We could add some constraint-based scheduling, to at minimum check for a > desired attribute (like HDFS-DataNode), or handle more complex constraints > like e.g. Marathon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MYRIAD-263) Add HDFS as system service in DC/OS vagrant testbed
[ https://issues.apache.org/jira/browse/MYRIAD-263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435155#comment-16435155 ] Juan Pedro commented on MYRIAD-263: --- +1 > Add HDFS as system service in DC/OS vagrant testbed > --- > > Key: MYRIAD-263 > URL: https://issues.apache.org/jira/browse/MYRIAD-263 > Project: Myriad > Issue Type: Task > Components: Vagrant >Reporter: Javi Roman >Priority: Major > > The local development environment based on vagrant-libvirt (DC/OS) needs > deploy HDFS as foundation for Myriad. We have available HDFS as Mesos > framework at Mesosphere Universe, however running this framework is not > straightforward. So I propose deploy HDFS as systemd services, this is easy > and allow us focus the in our development (Myriad). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MYRIAD-259) Reconciliation in register endpoint of the mesos scheduler
Juan Pedro created MYRIAD-259: - Summary: Reconciliation in register endpoint of the mesos scheduler Key: MYRIAD-259 URL: https://issues.apache.org/jira/browse/MYRIAD-259 Project: Myriad Issue Type: Improvement Components: Scheduler Reporter: Juan Pedro There are certain casuistics in which a newly-chosen master can send events to the register endpoint to a framework that is already registered. At this time it is necessary to perform a reconciliation so that the scheduler and the master do not have different views of the cluster. At this time it is not contemplated. The solution would be to perform a reconciliation in the process of registering or redirecting the event to the reregister endpoint. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
