Re: what are the tehcniques to automatically detect changes in Multiple DBs and automatically push them into Ignite Cache

[Juan Rodríguez Hortalá]

That looks like something you could do with Kafka connect
https://www.confluent.io/product/connectors/ using the jdbc source and the
ignite sink. Just my 2 cents

On Mon, Aug 6, 2018, 07:19 Deepa Kolwalkar  wrote:

> Thanks Prasad for your suggestions.
>
> The Legacy sytems are on different platforms and some of them are products
> .. so there is no way of implementing any custom logic in such products to
> send update messages. The Legacy Systems remain a black-box to us with only
> its DB which is accessible for viewing..
>
> Regards
>
>
>
>
>
> From:"Prasad Bhalerao" 
> To:user@ignite.apache.org
> Date:06-08-2018 16:51
> Subject:Re: what are the tehcniques to automatically detect
> changes in Multiple DBs and automatically push them into Ignite Cache
> --
>
>
>
> Can this back office legacy system send you a DB update message or can you
> make this back office system to send you DB update message?
>
> If yes then you can have the Id/primary key, DB operation and table name
> in this DB update message.
>
> In your application you use this information to refresh your cache using
> read through mechanism.
>
> Thanks,
> Prasad
>
> On Mon, Aug 6, 2018, 3:02 PM Deepa Kolwalkar <*deepa.kolwal...@tcs.com*
> > wrote:
> Thanks Denis.
>
> But as I mentioned in earlier mail, the Caches are meant to be Read-Only
> (only to be used by Microservices for fetching data).
> The Databases are updated by backoffice legacy systems.  Hence we cannot
> do a Write-through to the DBs via the CacheStore API.
>
> If anyone has used the Gridgain GoldenGate Adapter, then we would be glad
> to hear about any challenges/short-comings if any.
>
> Regards
>
>
>
> From:Denis Mekhanikov <*dmekhani...@gmail.com*
> >
> To:*user@ignite.apache.org* 
> Date:06-08-2018 13:18
> Subject:Re: what are the tehcniques to automatically detect
> changes in Multiple DBs and automatically push them into Ignite Cache
> --
>
>
>
> There is no such feature in Ignite.
> If you know, how to subscribe for events in the external database, then
> you can implement this logic yourself.
> You just need to perform put into Ignite cache for every insert into the
> external DB.
>
> But the recommended way to do it is to perform writing on Ignite.
> Cache store with *write-through*
> 
>  enabled
> will take care of writing the data into the external DB.
>
> Denis
>
> вс, 5 авг. 2018 г. в 17:32, Deepa Kolwalkar <*deepa.kolwal...@tcs.com*
> >:
> We have a requirement where Changes to data from Multiple DBs need to be
> periodically & automatically Pushed (not sure how) into various Ignite
> Caches
> Once the Data is available in the Ignite Caches, it will be persisted
> using Ignite Native Persistence, so that in the event of a crash, the Data
> can be loaded from Native Persistence.
> The Caches will be used in read-only manner by Clients (Microservices) .
>
> What is the Best technique for having Changes to data from Multiple DBs to
> be automatically put into the Ignite Caches ?
>
> While searching for this solution i came across this link :
>
> *http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html*
> 
> which suggests the following :
> ==
> Ignite does not provide such integration out of the box, however there
> a commercial offering from GridGain for that:
> *https://docs.gridgain.com/docs/goldengate-replication*
> 
> ==
>
> Was wondering whether we still need to use GoldenGate for such
> replications OR whether newer versions of Ignite are now supporting such
> asynchronous sync-ups with underlying DB changes
>
> Thanks
> =-=-=
> Notice: The information contained in this e-mail
> message and/or attachments to it may contain
> confidential or privileged information. If you are
> not the intended recipient, any dissemination, use,
> review, distribution, printing or copying of the
> information contained in this e-mail message
> and/or attachments to it are strictly prohibited. If
> you have received this communication in error,
> please notify us by reply e-mail or telephone and
> immediately and permanently delete the message
> and any attachments. Thank you
>

Re: Force flush of IGFS to secondary file system in DUAL_ASYNC mode

[Juan Rodríguez Hortalá]

Hi llya,

Thanks a lot for the detailed answer. It's nice to know there is a clear
path to achieve that flush.

Greetings,

Juan

On Mon, Jan 8, 2018 at 4:33 AM, ilya.kasnacheev 
wrote:

> Hello!
>
> After reviewing IGFS code, I think that you can do the following:
>
> You should save all file paths that are being migrated, and then call
> await(collectionWithAllFilePaths) on IgfsImpl. If it's a huge number of
> files, I imagine you can do this in batches.
>
> It will do the same synchronous wait that DUAL_SYNC would do, just from a
> different entry point. After await() returns you are safe to close IgfsImpl
> and shutdown your cluster.
>
> Note that I would like to have the same behaviour for
> IgfsImpl.close(cancel:
> false), but it's NOT there yet. I have filed
> https://issues.apache.org/jira/browse/IGNITE-7356 - do not hesitate to
> comment.
>
> Regards,
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: IGNITE-4862

[Juan Rodríguez Hortalá]

Hi Illya,

I have compiled the source for 2.3 after applying that patch, and now I can
use IGFS configured with values like   and , and I have been able to run some
benchmarks without exceptions or errors.

Thanks a lot for your help.

Greetings,

Juan



On Mon, Dec 18, 2017 at 5:02 AM, Ilya Kasnacheev 
wrote:

> Hello Juan!
>
> IGNITE-4862  was just
> merged into master branch, which means it will be in 2.4. You can also
> apply the commit yourself if you're building from sources.
>
> 5cd3e263b9ca1464044c1eec24fc642873a881ab is commit's sha.
>
> Regards,
> Ilya.
>
> --
> Ilya Kasnacheev
>
> 2017-12-13 16:05 GMT+03:00 ilya.kasnacheev :
>
>> Hello!
>>
>> I will make sure this ticket gets merged after tests, since its code seems
>> to be OK.
>>
>> This way it will make into the next version, in the meantime you can try
>> the
>> patch yourself, write your feedback if something goes wrong.
>>
>> Regards,
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
>

Force flush of IGFS to secondary file system in DUAL_ASYNC mode

[Juan Rodríguez Hortalá]

Hi,

When using IGFS with a secondary file system, with write behind configured
by using DUAL_ASYNC IgfsMode, is there any way to force the flush of the
data from the Ignite caches into the secondary file system? A possible
scenario here might be a temporary cluster with Ignite installed, that uses
IGFS with DUAL_ASYNC to write to an HDFS cluster running in a permanent
cluster that is configured as the secondary file system. In order to be
able to shutdown this cluster we need to know that all the data has been
flushed to HDFS or we might have data loss. For what I see in
http://apache-ignite-users.70518.x6.nabble.com/Flush-the-cache-into-the-persistence-store-manually-td5077.html
this wasn't available at the time that question was answered. The solution
proposed there seems to be traversing the cache writing each cached entry
to the data store that is cached. But for IGFS I understand that is not so
straightforward, because the dataCache and metadataCache used by IGFS don't
store the HDFS files directly, but the result of splitting them into
pieces.

Is there any way to flush the data from IGFS into HDFS? If not, is there
any recommendation about how we could traverse the dataCache and
metadataCache used by IGFS to manually write the data into HDFS? If we do
that traversal, is there any way to avoid the async writes of IGFS and the
write done in that traversal to interfere with each other, or lead to
duplicate writes?

Thanks a lot for your help!

Juan Rodriguez Hortala

Understanding configurations for IGFS

[Juan Rodríguez Hortalá]

Hi,

I'm trying to understand the configuration parameters for IGFS. My use case
is using IGFS with a secondary file system, thus acting as a cache for a
hadoop file system, without having to modify any existing application (just
the input and output path that will now use the igfs scheme). In the
javadoc for FileSystemConfiguration I see:

int getPerNodeBatchSize()
Gets number of file blocks buffered on local node before sending batch to
remote node.
int getPerNodeParallelBatchCount()
Gets number of batches that can be concurrently sent to remote node.
int getPrefetchBlocks()
Get number of pre-fetched blocks if specific file's chunk is requested.

What is the remote node here? I understand this doesn't have to do with
other ignite nodes holding backup copies, as that would be set in the cache
configuration.

I have also taken a look to http://apache-ignite-users.70518.x6.nabble.
com/IGFS-Data-cache-size-td2875.html but that post seems to refer to a
deprecated field FileSystemConfiguration.maxSpaceSize that I haven't been
able to see neither in the javadoc or in https://github.com/apache/
ignite/blob/2.3.0/modules/core/src/main/java/org/apache/
ignite/configuration/FileSystemConfiguration.java. Other questions that I
have regarding Ignite configuration in the context of this use case:

 - When I use ATOMIC for the atomicityMode of metaCacheConfiguration I get
an launch exception  "Failed to start grid: IGFS metadata cache should be
transactional: igfs". So I understand TRANSACTIONAL is required for
metaCacheConfiguration, but I get no error when using ATOMIC
for dataCacheConfiguration, is there any reason to use TRANSACTIONAL for
dataCacheConfiguration? I understand ATOMIC gets better performance if you
don't use the transaction features.

 - The readThrough, writeThrough,writeBehind fields for the
CacheConfiguration dataCacheConfiguration and metaCacheConfiguration have
any effect? Or maybe IGFS is setting them according to the IgfsMode
configured in the defaultMode field of FileSystemConfiguration?

- Similarly, does the setExpiryPolicyFactory in dataCacheConfiguration and
metaCacheConfiguration have any effect? I'd be interested in
using DUAL_ASYNC defaultMode, and I though that maybe the ExpiryPolicy
could give an upper bound for the time it takes for a record to be written
to the secondary file system, because it has been expired from the cache.
That way I could safely tear down the IGFS cluster after that time without
any data loss. Is there some way of achieving that? Otherwise I think
DUAL_ASYNC could only be used in long lived cluster, because I understand
there is no functionality to flush the IGFS caches into the secondary file
system.

- Similarly, does the eviction policy configured for dataCacheConfiguration
and metaCacheConfiguration have any effect? In any case I understand that
IGFS can never fail due to having no more space in the caches, because it
will evict the requires entries, saving them to the secondary file system
if needed in order to avoid data loss.

It would be nice if someone could point me to some webminar or
documentation specific for IGFS. I have already watched
https://www.youtube.com/watch?v=pshM_gy7Wig and I think it is a good
introduction, but I would like to get more details. I have also read the
book "High-Performance In-Memory Computing With Apache Ignite"

Thanks a lot for all your help.

Best Regards,

Juan

Re: ClassNotFoundException when using IgniteHadoopIgfsSecondaryFileSystem

[Juan Rodríguez Hortalá]

Hi Alexey,

Thanks for your answers. I was finally able to fix this, I was missing some
additional jars in the lib directory of the ignite installation in the
worker nodes. The exception thrown was
by org.apache.hadoop.conf.Configuration.getClassByName was a bit misleading
because it was referring to the file system class, while the class that was
missing in the classpath was a different one. It is working fine now.

Thanks a lot for your help!

On Thu, Dec 7, 2017 at 3:24 AM, Alexey Kukushkin 
wrote:

> I just replied you on another thread
> 
> with details on what you need to do to connect to a remote IGFS cluster.
> You still need Ignite configuration and Ignite-Hadoop dependencies on your
> spark nodes to connect to remote IGFS. You also need to add some Hadoop
> properties to make it find IGFS configuration.
>

Re: ClassNotFoundException when using IgniteHadoopIgfsSecondaryFileSystem

[Juan Rodríguez Hortalá]

Hi,

Thanks for your answer. I have both libs/ignite-core and
libs/ignite-hadoop/ignite-hadoop, and also the jar for the other hadoop
file system, in the list of jars visible on the java.class.path section
that is returned by the config command of the Ignite Visor. Is the order of
the jars relevant here? Are jars expected in any particular order? I also
added the jars for ignite-core, ignite-hadoop and ignite-shmem to
HADOOP_CLASSPATH in /etc/hadoop/conf/hadoop-env.sh.

This is working with , but it fails when I use
another uri with the scheme for the other hadoop file system. So I think
the problem is not finding the ignite jars, but the jars for the other
hadoop file system. For launching the IGFS daemons I use:

```
source /etc/default/hadoop # gets HADOOP_HOME
source /etc/hadoop/conf/hadoop-env.sh # gets IGNITE_HOME and
HADOOP_CLASSPATH
export USER_LIBS=${HADOOP_CLASSPATH}
${IGNITE_HOME}/bin/ignite.sh ignite-conf.xml
```

I'm not really sure about the role of ${USER_LIBS} here. Any other ideas?
Maybe some pointer to some tutorial about using
IgniteHadoopIgfsSecondaryFileSystem with a hadoop compatible file system
different to HDFS?

Thanks again,

Juan


On Wed, Dec 6, 2017 at 3:24 AM, Alexey Kukushkin 
wrote:

> Hi,
>
> I do not see any other reasons for a "class not found" exception but the
> class is not on the CLASSPATH. Can you make sure "ignite-core-xxx.jar" and
> "ignite-hadoop-xxx.jar" are on the hadoop's CLASSPATH.
>

Re: How IGFS URIs work

[Juan Rodríguez Hortalá]

Hi again,

I'm using the following ipcEndpointConfiguration in all nodes running ignite

  







Then from a node that is not running ignite I can do  `hadoop fs -ls
igfs://igfs@ip-10-0-0-85.ec2.internal:10500/` and it works just fine. But
my question is, if I use `igfs://igfs@ip-10-0-0-85.ec2.internal:10500/`
from a group of nodes that don't run IGFS locally, will all them connect to
the same node ip-10-0-0-85.ec2.internal? How can I distribute the load
among all nodes of the IGFS cluster, when reading from another group of
nodes than don't have IGFS installed locally?

Thanks,

Juan


On Wed, Dec 6, 2017 at 10:37 AM, Alexey Kukushkin  wrote:

> That is right - you cannot use SHMEM connection type if you have no local
> Ignite. SHMEM means "shared memory" and requires local Ignite.
>
> Change IGFS endpoint type to TCP (below replace __HOST__ and __PORT__ with
> IP and port of any host running Ignite):
>
> 
> 
> 
> 
> 
> 
> 
>
>

Re: How IGFS URIs work

[Juan Rodríguez Hortalá]

But if I do that from a compute node that doesn't have ignite, then I fail
to connect to the IGFS cluster:

[hadoop@ip-10-0-0-242 ~]$ hadoop fs -ls igfs://igfs@/
ls: Failed to communicate with IGFS: Failed to connect to IGFS
[endpoint=igfs://igfs@, attempts=[[type=SHMEM, port=10500,
err=java.io.IOException: Failed to connect shared memory endpoint to port
(is shared memory server endpoint up and running?): 10500], [type=TCP,
host=127.0.0.1, port=10500, err=java.io.IOException: Failed to connect to
endpoint [host=127.0.0.1, port=10500]]] (ensure that IGFS is running and
have IPC endpoint enabled; ensure that ignite-shmem-1.0.0.jar is in Hadoop
classpath if you use shared memory endpoint).
[hadoop@ip-10-0-0-242 ~]$

When I specify a node that is part of the IGFS cluster this works fine, for
example `hadoop fs -ls igfs://igfs@ip-10-0-0-85.ec2.internal:10500/`. Also,
`hadoop fs -ls igfs://igfs@/` works ok when I run it from the host
`ip-10-0-0-85`. I think this makes sense because if I don't specify a host
then localhost is used as default as specified in
https://apacheignite-fs.readme.io/docs/file-system#section-file-system-uri.
But as the ignite cluster is not running in the compute nodes then I cannot
use localhost as an address for them. Using igfs://igfs@/, therefore
localhost, would work for a typical Hadoop setting, where I have a YARN
cluster co-located with an Ignite cluster, just like we would do with HDFS.
But in settings like EMR, it is usual to have an instance group of core
nodes running both HDFS and YARN (both storage and compute), and a another
instance group of task nodes running only YARN (just compute, see
https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-instances.html
for details). It is possible to configure a IGFS URI that access a IGFS
cluster that is not running in the client node, but that balances the
requests among all nodes of the IGFS cluster?

Also, even if I have ignite running in all nodes, for example a cluster
with 3 slave nodes n1, n2, n3 running both the IGFS daemon and the YARN
Node Manager, if the IGFS daemon in node n1 crashes I would like to be able
to connect to the IGFS cluster from any YARN container running in n1,
because the IGFS cluster would still have 2 nodes available. That would be
analogous to what would happen with HDFS, where a failure in the data node
process running in n1 wouldn't prevent containers running in n1 to connect
to HDFS, because the HDFS URI is in fact referring to the Name Node, not to
n1. If I use igfs://igfs@/, which is the same as igfs://igfs@localhost/ to
connect to IGFS from all YARN containers running in n1, then I understand I
that if the IGFS daemon running in n1 crashes then I would lose connection
to IGFS from all YARN containers running in n1. That is not exactly a
single point of failure, but this would have a significant impact. Can you
clarify what would happen?

To summarize:
  - it is possible to configure a IGFS URI that access a IGFS cluster that
is not running in the client node, but that balances the requests among all
nodes of the IGFS cluster?
  - if I connect to IGFS from a client using igfs://igfs@/ and the local
IGFS daemon dies, does the client lose connection to all the IGFS cluster?,
even though other IGFS nodes are still alive, and I have their address
published in a node discovery service? (e.g. through
TcpDiscoveryZookeeperIpFinder)

Thanks again,

Juan


On Wed, Dec 6, 2017 at 3:42 AM, Alexey Kukushkin 
wrote:

> You do not need to specify the "host" part of the Hadoop URI at all if you
> do not want to limit processing to that specific host. Just specify
> "igfs://myIgfs@/" and Hadoop will load-balance the load for you across
> all IGFS nodes.
>

How IGFS URIs work

[Juan Rodríguez Hortalá]

Hi,

According to
https://apacheignite-fs.readme.io/docs/file-system#section-file-system-uri,
a URI for connecting to a ignite file system from a Hadoop compatible
application follows the pattern igfs://myIgfs@myHost:12345/. Let's assume I
have a ignite cluster with 5 nodes running in 5 different hosts data_host1,
data_host2, data_host3, data_host4, data_host5 in port 12345, and I have a
different set of 5 hosts compute_host1, compute_host2, compute_host3,
compute_host4, compute_host5 where I'm running some application that
connects to that IGFS cluster, for example a Spark program running on YARN
containers in the compute hosts. If I use the URI
igfs://myIgfs@data_host1:12345
in the Spark driver, and use that URI for all the Spark tasks running in
the compute hosts, will all the compute hosts connect to the same host
data_host1? If that is the case, then I would need to manually implement
some kind of load balancing in the application, in order to avoid hot spots
in Ignite. But maybe Ignite is treating the node in the URI as a kind of
seed node that is only used to get the set of cluster members, and
implements load balancing internally. Can you clarify which is the actual
behaviour? Any pointer to the documentation describing this would be much
appreciated.

Thanks a lot.

Juan Rodriguez Hortala

ClassNotFoundException when using IgniteHadoopIgfsSecondaryFileSystem

[Juan Rodríguez Hortalá]

Hi,

I'm trying to use IgniteHadoopIgfsSecondaryFileSystem on EMR, and I have
been able to make this work using HDFS in the uri. But when I use a
different scheme for another file system that is working ok with `hadoop
fs` and other applications, I get

```
[hadoop@ip-10-0-0-242 ~]$  hadoop fs -ls
igfs://igfs@ip-10-0-0-85.ec2.internal:10500/
ls: Generic IGFS error occurred.


and then in the node ip-10-0-0-85.ec2.internal I see in the logs, at the
same time the request is made, and exception `class
org.apache.ignite.IgniteException: java.lang.ClassNotFoundException` and
`Class not found` for the class implementing Hadoop FileSystem that is
extending org.apache.hadoop.fs.FileSystem that is associated to that scheme
in core-site.xml.

Before I added the path to core-site.xml in configPaths for
CachingHadoopFileSystemFactory I was getting
"org.apache.ignite.IgniteException: No FileSystem for scheme". I have tried
running setup-hadoop.sh  in that slave, and that creates some symlinks, but
I have the same error. The weird thing here is that when I check the node
config with ignite visor I see the jar that contains this class listed
in java.class.path.

Any idea how might I keep debugging this?

Thanks a lot in advance.

Juan

Re: Error running ignite in YARN

[Juan Rodríguez Hortalá]

Hi Ilya,

Thanks a lot for your help, I'll try and the test the patch.

Regarding how I run this, I wait for the cluster to enter in WAITING state
and then I ssh to the master node as described in
https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-connect-master-node-ssh.html,
and then I just run the commands there. Before running the command I put
the files in HDFS. We cannot use a bootstrap action here because they run
before HDFS starts, but another option is using a EMR step for this.

For the node manager logs, I follow
https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-web-interfaces.html
to have access to the resource manager UI, and then I see the nodes that
are involved with the YARN application. Then I ssh to some of the slave
nodes involved with the job from the EMR master, and check the node manager
logs in /var/log/hadoop-yarn. Also we could configure yarn like suggested
in https://slider.incubator.apache.org/docs/getting_started.html to have 1
hour of timeout before cleaning up containers, using
https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps.html
to configure that on EMR. The container files are then available in the EMR
slaves at /mnt/yarn/


I'll tell you how it goes when I have tested the patch.

Thanks a lot!

Juan


On Fri, Dec 1, 2017 at 6:43 AM, ilya.kasnacheev 
wrote:

> Hello once more!
>
> I have prepared a patch. I think it won't be hard for you to try it out
> since you are accustomized to building Ignite. Please try it, tell if it
> solves the current issue.  ignite-7080.patch
>  t1312/ignite-7080.patch>
>
> I was trying to set Ignite up on AWS EMR today, without much success. I
> wonder if you could provide me with some directions.
>
> > IGNITE_YARN_JAR=/mnt/ignite/apache-ignite-2.3.0-src/
> modules/yarn/target/ignite-yarn-2.3.0.jar
> >  yarn jar ${IGNITE_YARN_JAR} ${IGNITE_YARN_JAR}
> > /mnt/ignite/ignite_yarn.properties
>
> Can you tell me what were you using to launch this command? I have tried
> executing "Custom JAR" step on EMR cluster, after copying Ignite properties
> and JAR to /mnt on master manually. So I was informed that an application
> was started, and is now running, but no indication about execution results.
> I suspect it hung with the same problem that you outlined (or earlier), but
> I haven't found a way to view application logs, neither "node manager" logs
> that you are referring to. Can you please point me where I can watch
> intermediate execution results?
>
> Also there's two data access questions. First, Ignite YARN jar expects two
> files on local FS, the jar itself, and properties file. Is there a way to
> make sure that these files are on file system, at predictable locations,
> when starting EMR cluster? To be able to add "Custom JAR" step to cluster
> initialization? E.g. specify an s3 bucket to be mounted to master node?
>
> Another question is that you are specifying ignite zip and ignite config
> are
> on hdfs. How would you put these files to hdfs when cluster is created? Or,
> for that matter, after it is created? I haven't found any access to HDFS in
> EMR console, neither any instructions on how to access it.
>
> Maybe we should change this process in order to use Ignite YARN with EMR
> without extensive boilerplate?
>
> Regards,
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Error running ignite in YARN

[Juan Rodríguez Hortalá]

Hi,

Any thoughts on this? Do you need any further details about the setup?

Thanks,

Juan

On Tue, Nov 21, 2017 at 8:59 AM, Juan Rodríguez Hortalá <
juan.rodriguez.hort...@gmail.com> wrote:

> Hi,
>
> Anyone might help a newbie ramping up with Ignite on YARN?
>
> Thanks,
>
> Juan
>
>
> On Sun, Nov 19, 2017 at 7:34 PM, Juan Rodríguez Hortalá <
> juan.rodriguez.hort...@gmail.com> wrote:
>
>> Hi,
>>
>> I'm trying to run ignite on AWS EMR as a YARN application, using
>> zookeeper for node discovery. I have compiled ignite with
>>
>> ```
>> mvn clean package -DskipTests -Dignite.edition=hadoop
>> -Dhadoop.version=2.7.3
>> ```
>>
>> I'm using ignite_yarn.properties
>>
>> ```
>> # The number of nodes in the cluster.
>> IGNITE_NODE_COUNT=3
>>
>> # The number of CPU Cores for each Apache Ignite node.
>> IGNITE_RUN_CPU_PER_NODE=1
>>
>> # The number of Megabytes of RAM for each Apache Ignite node.
>> IGNITE_MEMORY_PER_NODE=500
>>
>> IGNITE_PATH=hdfs:///user/hadoop/ignite/apache-ignite-2.3.0-
>> hadoop-2.7.3.zip
>>
>> IGNITE_XML_CONFIG=hdfs:///user/hadoop/ignite/ignite_conf.xml
>>
>> # Local path
>> IGNITE_WORK_DIR=/mnt
>>
>> # Local path
>> IGNITE_RELEASES_DIR=/mnt
>>
>> IGNITE_WORKING_DIR=/mnt
>> 
>>
>> and ignite_conf.xml as
>>
>> ```
>> 
>> http://www.springframework.org/schema/beans;
>>xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>>xsi:schemaLocation="
>> http://www.springframework.org/schema/beans
>> http://www.springframework.org/schema/beans/spring-beans.xsd;>
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>>
>> 
>> 
>> 
>>   
>>   
>>
>>   > value="ip-10-0-0-173.ec2.internal:2181"/>
>>   
>>   
>> 
>> 
>> 
>>   
>> 
>> > value="config/ignite-log4j2.xml"/>
>>   
>> 
>> 
>> 
>> ```
>>
>> Then I launch the yarn job as
>>
>>
>> ```
>> IGNITE_YARN_JAR=/mnt/ignite/apache-ignite-2.3.0-src/modules/
>> yarn/target/ignite-yarn-2.3.0.jar
>>  yarn jar ${IGNITE_YARN_JAR} ${IGNITE_YARN_JAR}
>> /mnt/ignite/ignite_yarn.properties
>> ```
>>
>> The app launches and the application master is outputting logs, but
>> containers only last some seconds running, and the application is
>> constantly asking for more containers. For example, in the application
>> master log
>>
>> ```
>>
>> Nov 20, 2017 3:08:30 AM org.apache.ignite.yarn.ApplicationMaster 
>> onContainersAllocated
>> INFO: Launching container: container_1511142795395_0005_01_017079.
>> 17/11/20 03:08:30 INFO impl.ContainerManagementProtocolProxy: Opening proxy 
>> : ip-10-0-0-230.ec2.internal:8041
>> Nov 20, 2017 3:08:30 AM org.apache.ignite.yarn.ApplicationMaster 
>> onContainersAllocated
>> INFO: Launching container: container_1511142795395_0005_01_017080.
>> 17/11/20 03:08:30 INFO impl.ContainerManagementProtocolProxy: Opening proxy 
>> : ip-10-0-0-78.ec2.internal:8041
>> Nov 20, 2017 3:08:30 AM org.apache.ignite.yarn.ApplicationMaster 
>> onContainersAllocated
>> INFO: Launching container: container_1511142795395_0005_01_017081.
>> 17/11/20 03:08:30 INFO impl.ContainerManagementProtocolProxy: Opening proxy 
>> : ip-10-0-0-193.ec2.internal:8041
>> Nov 20, 2017 3:08:31 AM org.apache.ignite.yarn.ApplicationMaster 
>> onContainersCompleted
>> INFO: Container completed. Container id: 
>> container_1511142795395_0005_01_017080. State: COMPLETE.
>> Nov 20, 2017 3:08:31 AM org.apache.ignite.yarn.ApplicationMaster 
>> onContainersCompleted
>> INFO: Container completed. Container id: 
>> container_1511142795395_0005_01_017081. State: COMPLETE.
>> Nov 20, 2017 3:08:31 AM org.apache.ignite.yarn.ApplicationMaster 
>> onContainersCompleted
>> INFO: Container completed. Container id: 
>> container_1511142795395_0005_01_017079. State: COMPLETE.
>> Nov 20, 2017 3:08:31 AM org.apache.ignite.yarn.ApplicationMaster 
>> onContainersAllocated
>>
>

Re: Error running ignite in YARN

[Juan Rodríguez Hortalá]

Hi,

Anyone might help a newbie ramping up with Ignite on YARN?

Thanks,

Juan


On Sun, Nov 19, 2017 at 7:34 PM, Juan Rodríguez Hortalá <
juan.rodriguez.hort...@gmail.com> wrote:

> Hi,
>
> I'm trying to run ignite on AWS EMR as a YARN application, using zookeeper
> for node discovery. I have compiled ignite with
>
> ```
> mvn clean package -DskipTests -Dignite.edition=hadoop
> -Dhadoop.version=2.7.3
> ```
>
> I'm using ignite_yarn.properties
>
> ```
> # The number of nodes in the cluster.
> IGNITE_NODE_COUNT=3
>
> # The number of CPU Cores for each Apache Ignite node.
> IGNITE_RUN_CPU_PER_NODE=1
>
> # The number of Megabytes of RAM for each Apache Ignite node.
> IGNITE_MEMORY_PER_NODE=500
>
> IGNITE_PATH=hdfs:///user/hadoop/ignite/apache-ignite-2.
> 3.0-hadoop-2.7.3.zip
>
> IGNITE_XML_CONFIG=hdfs:///user/hadoop/ignite/ignite_conf.xml
>
> # Local path
> IGNITE_WORK_DIR=/mnt
>
> # Local path
> IGNITE_RELEASES_DIR=/mnt
>
> IGNITE_WORKING_DIR=/mnt
> 
>
> and ignite_conf.xml as
>
> ```
> 
> http://www.springframework.org/schema/beans;
>xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>xsi:schemaLocation="
> http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd;>
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>
> 
> 
> 
>   
>   
>
>value="ip-10-0-0-173.ec2.internal:2181"/>
>   
>   
> 
> 
> 
>   
> 
>  value="config/ignite-log4j2.xml"/>
>   
> 
> 
> 
> ```
>
> Then I launch the yarn job as
>
>
> ```
> IGNITE_YARN_JAR=/mnt/ignite/apache-ignite-2.3.0-src/
> modules/yarn/target/ignite-yarn-2.3.0.jar
>  yarn jar ${IGNITE_YARN_JAR} ${IGNITE_YARN_JAR} /mnt/ignite/ignite_yarn.
> properties
> ```
>
> The app launches and the application master is outputting logs, but
> containers only last some seconds running, and the application is
> constantly asking for more containers. For example, in the application
> master log
>
> ```
>
> Nov 20, 2017 3:08:30 AM org.apache.ignite.yarn.ApplicationMaster 
> onContainersAllocated
> INFO: Launching container: container_1511142795395_0005_01_017079.
> 17/11/20 03:08:30 INFO impl.ContainerManagementProtocolProxy: Opening proxy : 
> ip-10-0-0-230.ec2.internal:8041
> Nov 20, 2017 3:08:30 AM org.apache.ignite.yarn.ApplicationMaster 
> onContainersAllocated
> INFO: Launching container: container_1511142795395_0005_01_017080.
> 17/11/20 03:08:30 INFO impl.ContainerManagementProtocolProxy: Opening proxy : 
> ip-10-0-0-78.ec2.internal:8041
> Nov 20, 2017 3:08:30 AM org.apache.ignite.yarn.ApplicationMaster 
> onContainersAllocated
> INFO: Launching container: container_1511142795395_0005_01_017081.
> 17/11/20 03:08:30 INFO impl.ContainerManagementProtocolProxy: Opening proxy : 
> ip-10-0-0-193.ec2.internal:8041
> Nov 20, 2017 3:08:31 AM org.apache.ignite.yarn.ApplicationMaster 
> onContainersCompleted
> INFO: Container completed. Container id: 
> container_1511142795395_0005_01_017080. State: COMPLETE.
> Nov 20, 2017 3:08:31 AM org.apache.ignite.yarn.ApplicationMaster 
> onContainersCompleted
> INFO: Container completed. Container id: 
> container_1511142795395_0005_01_017081. State: COMPLETE.
> Nov 20, 2017 3:08:31 AM org.apache.ignite.yarn.ApplicationMaster 
> onContainersCompleted
> INFO: Container completed. Container id: 
> container_1511142795395_0005_01_017079. State: COMPLETE.
> Nov 20, 2017 3:08:31 AM org.apache.ignite.yarn.ApplicationMaster 
> onContainersAllocated
>
> ```
>
> In the logs for a node manager I see containers seem to fail when they are
> launched, because the corresponding bash command is not well formed
>
> ```
> 2017-11-20 03:08:47,810 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.container.ContainerImpl (AsyncDispatcher
> event handler): Container container_1511142795395_0005_01_017281
> transitioned from LOCALIZED to RUNNING
> 2017-11-20 03:08:47,811 INFO 
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor
> (ContainersLauncher #4): launchContainer: [bash, /mnt/yarn/usercache/hadoop/
> appcache/application_1511142795395_0005/container_
> 1511142795395_0005_01_017281/default_container_executor.sh]
> 2017-11-20 03:08

Error running ignite in YARN

[Juan Rodríguez Hortalá]

Hi,

I'm trying to run ignite on AWS EMR as a YARN application, using zookeeper
for node discovery. I have compiled ignite with

```
mvn clean package -DskipTests -Dignite.edition=hadoop -Dhadoop.version=2.7.3
```

I'm using ignite_yarn.properties

```
# The number of nodes in the cluster.
IGNITE_NODE_COUNT=3

# The number of CPU Cores for each Apache Ignite node.
IGNITE_RUN_CPU_PER_NODE=1

# The number of Megabytes of RAM for each Apache Ignite node.
IGNITE_MEMORY_PER_NODE=500

IGNITE_PATH=hdfs:///user/hadoop/ignite/apache-ignite-2.3.0-hadoop-2.7.3.zip

IGNITE_XML_CONFIG=hdfs:///user/hadoop/ignite/ignite_conf.xml

# Local path
IGNITE_WORK_DIR=/mnt

# Local path
IGNITE_RELEASES_DIR=/mnt

IGNITE_WORKING_DIR=/mnt


and ignite_conf.xml as

```

http://www.springframework.org/schema/beans;
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
   xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd;>
















  
  
   
  
  
  



  


  



```

Then I launch the yarn job as


```
IGNITE_YARN_JAR=/mnt/ignite/apache-ignite-2.3.0-src/modules/yarn/target/ignite-yarn-2.3.0.jar
 yarn jar ${IGNITE_YARN_JAR} ${IGNITE_YARN_JAR}
/mnt/ignite/ignite_yarn.properties
```

The app launches and the application master is outputting logs, but
containers only last some seconds running, and the application is
constantly asking for more containers. For example, in the application
master log

```

Nov 20, 2017 3:08:30 AM org.apache.ignite.yarn.ApplicationMaster
onContainersAllocated
INFO: Launching container: container_1511142795395_0005_01_017079.
17/11/20 03:08:30 INFO impl.ContainerManagementProtocolProxy: Opening
proxy : ip-10-0-0-230.ec2.internal:8041
Nov 20, 2017 3:08:30 AM org.apache.ignite.yarn.ApplicationMaster
onContainersAllocated
INFO: Launching container: container_1511142795395_0005_01_017080.
17/11/20 03:08:30 INFO impl.ContainerManagementProtocolProxy: Opening
proxy : ip-10-0-0-78.ec2.internal:8041
Nov 20, 2017 3:08:30 AM org.apache.ignite.yarn.ApplicationMaster
onContainersAllocated
INFO: Launching container: container_1511142795395_0005_01_017081.
17/11/20 03:08:30 INFO impl.ContainerManagementProtocolProxy: Opening
proxy : ip-10-0-0-193.ec2.internal:8041
Nov 20, 2017 3:08:31 AM org.apache.ignite.yarn.ApplicationMaster
onContainersCompleted
INFO: Container completed. Container id:
container_1511142795395_0005_01_017080. State: COMPLETE.
Nov 20, 2017 3:08:31 AM org.apache.ignite.yarn.ApplicationMaster
onContainersCompleted
INFO: Container completed. Container id:
container_1511142795395_0005_01_017081. State: COMPLETE.
Nov 20, 2017 3:08:31 AM org.apache.ignite.yarn.ApplicationMaster
onContainersCompleted
INFO: Container completed. Container id:
container_1511142795395_0005_01_017079. State: COMPLETE.
Nov 20, 2017 3:08:31 AM org.apache.ignite.yarn.ApplicationMaster
onContainersAllocated

```

In the logs for a node manager I see containers seem to fail when they are
launched, because the corresponding bash command is not well formed

```
2017-11-20 03:08:47,810 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl
(AsyncDispatcher event handler): Container
container_1511142795395_0005_01_017281 transitioned from LOCALIZED to
RUNNING
2017-11-20 03:08:47,811 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor
(ContainersLauncher #4): launchContainer: [bash,
/mnt/yarn/usercache/hadoop/appcache/application_1511142795395_0005/container_1511142795395_0005_01_017281/default_container_executor.sh]
2017-11-20 03:08:47,819 WARN
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor
(ContainersLauncher #4): Exit code from container
container_1511142795395_0005_01_017281 is : 2
2017-11-20 03:08:47,819 WARN
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor
(ContainersLauncher #4): Exception from container-launch with container ID:
container_1511142795395_0005_01_017281 and exit code: 2
ExitCodeException exitCode=2:
/mnt/yarn/usercache/hadoop/appcache/application_1511142795395_0005/container_1511142795395_0005_01_017281/launch_container.sh:
line 4: syntax error near unexpected token `('
/mnt/yarn/usercache/hadoop/appcache/application_1511142795395_0005/container_1511142795395_0005_01_017281/launch_container.sh:
line 4: `export BASH_FUNC_run_prestart()="() {  su -s /bin/bash $SVC_USER
-c "cd $WORKING_DIR && $EXEC_PATH --config '$CONF_DIR' start $DAEMON_FLAGS"'

at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
at org.apache.hadoop.util.Shell.run(Shell.java:479)