copyOnRead

2016-10-04 Thread styriver 
Hello I would like to confirm something that I believe I proved in test
cases. 

Cache configured as REPLICATED and ONHEAP_TIERED.

If I run as a client node and attempt to make an update I do not get the
reference but rather a copy of the value. If I run as a server node and
perform the update I get the reference. In my test case I fail when I run
this as a client node and attempt to update the cached value. it is
successful when running as a server node and updating the value.  In order
for the client node to work I need to perform a getAndUpdate.

Is this expected behavior?



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/copyOnRead-tp8097.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: spark SQL thriftserver over ignite and cassandra

2016-10-04 Thread Denis Magda 
Hi Vincent,

See my answers inline

> On Oct 4, 2016, at 12:54 AM, vincent gromakowski 
>  wrote:
> 
> Hi,
> I know that Ignite has SQL support but:
> - ODBC driver doesn't seem to provide HTTP(S) support, which is easier to 
> integrate on corporate networks with rules, firewalls, proxies

Igor Sapego, what URIs are supported presently? 

> - The SQL engine doesn't seem to scale like Spark SQL would. For instance, 
> Spark won't generate OOM is dataset (source or result) doesn't fit in memory. 
> From Ignite side, it's not clear…

OOM is not related to scalability topic at all. This is about application’s 
logic. 

Ignite SQL engine perfectly scales out along with your cluster. Moreover, 
Ignite supports indexes which allows you to get O(logN) running time complexity 
for your SQL queries while in case of Spark you will face with full-scans 
(O(N)) all the time.

However, to benefit from Ignite SQL queries you have to put all the data 
in-memory. Ignite doesn’t go to a CacheStore (Cassandra, relational database, 
MongoDB, etc) while a SQL query is executed and won’t preload anything from an 
underlying CacheStore. Automatic preloading works for key-value queries like 
cache.get(key).

> - Spark thrift can manage multi tenancy: different users can connect to the 
> same SQL engine and share cache. In Ignite it's one cache per user, so a big 
> waste of RAM.

Everyone can connect to an Ignite cluster and work with the same set of 
distributed caches. I’m not sure why you need to create caches with the same 
content for every user.

If you need a real multi-tenancy support where cacheA is allowed to be accessed 
by a group of users A only and cacheB by users from group B then you can take a 
look at GridGain which is built on top of Ignite
https://gridgain.readme.io/docs/multi-tenancy


> 
> What I want to achieve is :
> - use Cassandra for data store as it provides idempotence (HDFS/hive 
> doesn't), resulting in exactly once semantic without any duplicates. 
> - use Spark SQL thriftserver in multi tenancy for large scale adhoc analytics 
> queries (> TB) from an ODBC driver through HTTP(S) 
> - accelerate Cassandra reads when the data modeling of the Cassandra table 
> doesn't fit the queries. Queries would be OLAP style: target multiple C* 
> partitions, groupby or filters on lots of dimensions that aren't necessarely 
> in the C* table key.
> 

As it was mentioned Ignite uses Cassandra as a CacheStore. You should keep this 
in mind. Before trying to assemble all the chain I would recommend you trying 
to connect Spark SQL thrift server directly to Ignite and work with its shared 
RDDs [1]. A shared RDD (basically Ignite cache) can be backed by Cassandra. 
Probably this chain will work for you but I can’t give more precise guidance on 
this.

[1] https://apacheignite-fs.readme.io/docs/ignite-for-spark
 
—
Denis

> Thanks for your advises
> 
> 
> 2016-10-04 6:51 GMT+02:00 Jörn Franke  >:
> I am not sure that this will be performant. What do you want to achieve here? 
> Fast lookups? Then the Cassandra Ignite store might be the right solution. If 
> you want to do more analytic style of queries then you can put the data on 
> HDFS/Hive and use the Ignite HDFS cache to cache certain partitions/tables in 
> Hive in-memory. If you want to go to iterative machine learning algorithms 
> you can go for Spark on top of this. You can use then also Ignite cache for 
> Spark RDDs.
> 
> On 4 Oct 2016, at 02:24, Alexey Kuznetsov  > wrote:
> 
>> Hi, Vincent!
>> 
>> Ignite also has SQL support (also scalable), I think it will be much faster 
>> to query directly from Ignite than query from Spark.
>> Also please mind, that before executing queries you should load all needed 
>> data to cache.
>> To load data from Cassandra to Ignite you may use Cassandra store [1].
>> 
>> [1] https://apacheignite.readme.io/docs/ignite-with-apache-cassandra 
>> 
>> 
>> On Tue, Oct 4, 2016 at 4:19 AM, vincent gromakowski 
>> > wrote:
>> Hi,
>> I am evaluating the possibility to use Spark SQL (and its scalability) over 
>> an Ignite cache with Cassandra persistent store to increase read workloads 
>> like OLAP style analytics.
>> Is there any way to configure Spark thriftserver to load an external table 
>> in Ignite like we can do in Cassandra ?
>> Here is an example of config for spark backed by cassandra
>> 
>> CREATE EXTERNAL TABLE MyHiveTable 
>> ( id int, data string ) 
>> STORED BY 'org.apache.hadoop.hive.cassandra.cql.CqlStorageHandler' 
>> TBLPROPERTIES ("cassandra.host" = "x.x.x.x", "cassandra.ks.name 
>> " = "test" , 
>>   "cassandra.cf.name " = "mytable" , 
>>

Re: Ignite Cluster Communication with SSH Tunnels

2016-10-04 Thread Krzysztof 
On the 1.8 problem stays the same (I switched off by mistake 48500
before - I switched it on again now): node connects, is listed as part
of the cluster, discovery sends list of nodes with internal addresses
and they are not translated on the local node, so even the node is
listed as part of the cluster, exceptions appear that the client
cannot connect to any of the nodes from the send list...

Now it fails on

   ctx.io().send(node, new
GridDhtAffinityAssignmentRequest(key.get1(), key.get2()),
at GridDhtAssignmentFetchFuture:185
trying to connect to port 48500 of the first node from the
availableNodes list. Why is it connecting to discovery port, even it
had fetched the cluster info already?



[02:25:57,195][INFO ][main][IgniteKernal] Non-loopback local IPs:
192.168.0.102, fe80:0:0:0:6e40:8ff:fe91:c0a4%en0,
fe80:0:0:0:9485:8fff:fec1:9de8%awdl0

[02:25:57,195][INFO ][main][IgniteKernal] Enabled local MACs:
6C400891C0A4, 96858FC19DE8

[02:25:57,213][INFO ][main][IgnitePluginProcessor] Configured plugins:

[02:25:57,213][INFO ][main][IgnitePluginProcessor]   ^-- None

[02:25:57,213][INFO ][main][IgnitePluginProcessor]

[02:25:59,804][INFO ][main][TcpCommunicationSpi] Successfully bound to
TCP port [port=47099, locHost=localhost/127.0.0.1]

[02:26:00,451][WARN ][main][NoopCheckpointSpi] Checkpoints are
disabled (to enable configure any GridCheckpointSpi implementation)

[02:26:00,486][WARN ][main][GridCollisionManager] Collision resolution
is disabled (all jobs will be activated upon arrival).

[02:26:00,489][WARN ][main][NoopSwapSpaceSpi] Swap space is disabled.
To enable use FileSwapSpaceSpi.

[02:26:00,491][INFO ][main][IgniteKernal] Security status
[authentication=off, tls/ssl=off]

[02:26:00,776][INFO ][main][GridTcpRestProtocol] Command protocol
successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0,
port=11211]

[02:26:06,180][INFO ][main][GridCacheProcessor] Started cache
[name=ignite-sys-cache, mode=REPLICATED]

[02:26:06,194][INFO ][main][GridCacheProcessor] Started cache
[name=ignite-atomics-sys-cache, mode=PARTITIONED]

[02:26:06,214][INFO ][main][GridCacheProcessor] Started cache
[name=ignite-marshaller-sys-cache, mode=REPLICATED]

[02:26:22,657][WARN ][exchange-worker-#46%null%][TcpCommunicationSpi]
Connect timed out (consider increasing 'failureDetectionTimeout'
configuration property) [addr=/192.168.168.8:47100,
failureDetectionTimeout=15000]

[02:26:22,660][WARN ][exchange-worker-#46%null%][TcpCommunicationSpi]
Connect timed out (consider increasing 'failureDetectionTimeout'
configuration property) [addr=/127.0.0.1:47100,
failureDetectionTimeout=15000]

[02:26:22,661][WARN ][exchange-worker-#46%null%][TcpCommunicationSpi]
Failed to connect to a remote node (make sure that destination node is
alive and operating system firewall is disabled on local and remote
hosts) [addrs=[/192.168.168.8:47100, /127.0.0.1:47100,
0:0:0:0:0:0:0:1%lo:47100]]

[02:26:36,223][WARN ][main][GridCachePartitionExchangeManager] Failed
to wait for initial partition map exchange. Possible reasons are:

  ^-- Transactions in deadlock.

  ^-- Long running transactions (ignore if this is the case).

  ^-- Unreleased explicit locks.

[02:26:54,741][ERROR][exchange-worker-#46%null%][GridDhtAssignmentFetchFuture]
Failed to request affinity assignment from remote node (will continue
to another node): TcpDiscoveryNode
[id=337d89f0-c14a-4b5d-8d33-b995331dcf63, addrs=[0:0:0:0:0:0:0:1%lo,
127.0.0.1, 192.168.168.8], sockAddrs=[0:0:0:0:0

:0:0:1%lo:48500, /127.0.0.1:48500, /192.168.168.8:48500],
discPort=48500, order=218, intOrder=110,
lastExchangeTime=1475627164474, loc=false,
ver=1.8.0#20161004-sha1:a370bad1, isClient=false]

class org.apache.ignite.IgniteCheckedException: Failed to send message
(node may have left the grid or TCP connection cannot be established
due to firewall issues) [node=TcpDiscoveryNode
[id=337d89f0-c14a-4b5d-8d33-b995331dcf63, addrs=[0:0:0:0:0:0:0:1%lo,
127.0.0.1, 192.168.168.8], sockAddrs=[0:0:0:

0:0:0:0:1%lo:48500, /127.0.0.1:48500, /192.168.168.8:48500],
discPort=48500, order=218, intOrder=110,
lastExchangeTime=1475627164474, loc=false,
ver=1.8.0#20161004-sha1:a370bad1, isClient=false], topic=TOPIC_CACHE,
msg=GridDhtAffinityAssignmentRequest [topVer=AffinityTopologyVersion
[topVer=276, min

orTopVer=0]], policy=4]

at 
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1309)

at 
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1373)

at 
org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:841)

at 
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtAssignmentFetchFuture.requestFromNextNode(GridDhtAssignmentFetchFuture.java:185)

at 
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtAssignmentFetchFuture.init(GridDhtAssignmentFetchFuture.java:107)

at 
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.fetchAffinityOnJoin

Re: Question about rmvall command of Rest API

2016-10-04 Thread vkulichenko 
Hi Satoshi,

I still can't reproduce this. Can you show your cache configuration and
describe how you populate the data?

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Question-about-rmvall-command-of-Rest-API-tp7903p8094.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Accessing igfs file through java

2016-10-04 Thread vkulichenko 
Sateesh,

IgfsPath should be a "local" path, e.g. /ignite/fs. You work with IGFS as
with a regular file system, but the data will be stored in a distributed
fashion on the cluster.

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Accessing-igfs-file-through-java-tp8073p8092.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: spark SQL thriftserver over ignite and cassandra

2016-10-04 Thread vincent gromakowski 
Do you have any remark/correction on my  assumptions ?

Le 4 oct. 2016 9:54 AM, "vincent gromakowski" 
a écrit :

> Hi,
> I know that Ignite has SQL support but:
> - ODBC driver doesn't seem to provide HTTP(S) support, which is easier to
> integrate on corporate networks with rules, firewalls, proxies
> - The SQL engine doesn't seem to scale like Spark SQL would. For instance,
> Spark won't generate OOM is dataset (source or result) doesn't fit in
> memory. From Ignite side, it's not clear...
> - Spark thrift can manage multi tenancy: different users can connect to
> the same SQL engine and share cache. In Ignite it's one cache per user, so
> a big waste of RAM.
>
> What I want to achieve is :
> - use Cassandra for data store as it provides idempotence (HDFS/hive
> doesn't), resulting in exactly once semantic without any duplicates.
> - use Spark SQL thriftserver in multi tenancy for large scale adhoc
> analytics queries (> TB) from an ODBC driver through HTTP(S)
> - accelerate Cassandra reads when the data modeling of the Cassandra table
> doesn't fit the queries. Queries would be OLAP style: target multiple C*
> partitions, groupby or filters on lots of dimensions that aren't
> necessarely in the C* table key.
>
> Thanks for your advises
>
>
> 2016-10-04 6:51 GMT+02:00 Jörn Franke :
>
>> I am not sure that this will be performant. What do you want to achieve
>> here? Fast lookups? Then the Cassandra Ignite store might be the right
>> solution. If you want to do more analytic style of queries then you can put
>> the data on HDFS/Hive and use the Ignite HDFS cache to cache certain
>> partitions/tables in Hive in-memory. If you want to go to iterative machine
>> learning algorithms you can go for Spark on top of this. You can use then
>> also Ignite cache for Spark RDDs.
>>
>> On 4 Oct 2016, at 02:24, Alexey Kuznetsov 
>> wrote:
>>
>> Hi, Vincent!
>>
>> Ignite also has SQL support (also scalable), I think it will be much
>> faster to query directly from Ignite than query from Spark.
>> Also please mind, that before executing queries you should load all
>> needed data to cache.
>> To load data from Cassandra to Ignite you may use Cassandra store [1].
>>
>> [1] https://apacheignite.readme.io/docs/ignite-with-apache-cassandra
>>
>> On Tue, Oct 4, 2016 at 4:19 AM, vincent gromakowski <
>> vincent.gromakow...@gmail.com> wrote:
>>
>>> Hi,
>>> I am evaluating the possibility to use Spark SQL (and its scalability)
>>> over an Ignite cache with Cassandra persistent store to increase read
>>> workloads like OLAP style analytics.
>>> Is there any way to configure Spark thriftserver to load an external
>>> table in Ignite like we can do in Cassandra ?
>>> Here is an example of config for spark backed by cassandra
>>>
>>> CREATE EXTERNAL TABLE MyHiveTable
>>> ( id int, data string )
>>> STORED BY 'org.apache.hadoop.hive.cassandra.cql.CqlStorageHandler'
>>>
>>> TBLPROPERTIES ("cassandra.host" = "x.x.x.x", "cassandra.ks.name"
>>> = "test" ,
>>>   "cassandra.cf.name" = "mytable" ,
>>>   "cassandra.ks.repfactor" = "1" ,
>>>   "cassandra.ks.strategy" =
>>> "org.apache.cassandra.locator.SimpleStrategy" );
>>>
>>>
>>
>>
>> --
>> Alexey Kuznetsov
>>
>>
>

Re: Ignite Cluster Communication with SSH Tunnels

2016-10-04 Thread vkulichenko 
All nodes should run on the same version.

What is exactly different in log4j behavior?

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Cluster-Communication-with-SSH-Tunnels-tp273p8090.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Couchbase as persistent store

2016-10-04 Thread vkulichenko 
You have write behind enabled. This means that the data will be flushed
eventually. Does it work if you disable write behind (in this case the store
will be updated synchronously with the cache update)?

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Couchbase-as-persistent-store-tp7476p8089.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Performance issues with Ignite L2 cache compared to ehcache

2016-10-04 Thread Himetic 
Let's see...

1) we've got 5 hosts running the service, each of which has an ignite
instance.
2) we currently don't have any ignite-only nodes, just the ones embedded in
the service.
3)  spring-ignitecache.xml

  
4) 1.7.0
5) I was tasked with trying out a distributed cache to see if it would give
us better performance than the local ehcache we're using currently.  Because
of the way we do requests, as regards batching, querying by id, etc, it may
be that a distributed cache just doesn't actually make sense for us,
unfortunately.  Failure, however, is always an option.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Performance-issues-with-Ignite-L2-cache-compared-to-ehcache-tp8069p8086.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: How to stop ignite node gracefully

2016-10-04 Thread yucigou 
In addition to "kill -9", you can also use ignitevisorcmd.sh, and then use
the open command to connect Visor console to the grid. Now you can just type
"kill" to use the kill command in an interactive way, to stop/restart an
Ignite node, gracefully I suppose.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/How-to-stop-ignite-node-gracefully-tp7209p8085.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Support for SQL insert, update and delete queries in Apache Ignite(1.7.0)

2016-10-04 Thread Alexander Paschenko 
As I said, there's master based branch ignite-2294 in public repo. You
can try to use it. However, currently there's no mechanism that would
allow you to have binary classless keys in binary mode - either your
keys should be standard JDK/SQL types (numbers, strings, dates, UUID,
etc), or your server nodes must have classes for binary keys, or you
have to use non binary marshalling.
AFAIK, 1.8.0 is due November.

2016-10-04 16:45 GMT+03:00 rishi007bansod :
> Is there any temporary patch that I can get? what is tentative date of
> version 1.8.0 release?
>
>
>
> --
> View this message in context: 
> http://apache-ignite-users.70518.x6.nabble.com/Support-for-SQL-insert-update-and-delete-queries-in-Apache-Ignite-1-7-0-tp8078p8083.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Support for SQL insert, update and delete queries in Apache Ignite(1.7.0)

2016-10-04 Thread rishi007bansod 
Is there any temporary patch that I can get? what is tentative date of
version 1.8.0 release?



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Support-for-SQL-insert-update-and-delete-queries-in-Apache-Ignite-1-7-0-tp8078p8083.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Unexpected flag value

2016-10-04 Thread dmitry.parkhonin 
Hi Vladislav,

It was my mistake - the whole task class hierarchy was overcomplicated, with
callbacks. I have simplified it and everything works fine now.
Thanks for help,

Dmitry.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Unexpected-flag-value-tp8050p8082.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Ignite Cluster Communication with SSH Tunnels

2016-10-04 Thread Krzysztof 
Thanks for the hint!
I was trying to use BasicAddressResolver from 1.7.1, but it was never
called for the list returned by the discovery.
I reckon the whole cluster must be the same version? With 1.8 on the
client only, even discovery does not work, just stays in the endless
loop.
I will convert the full cluster (not only client outside the cluster) into 1.8.

I noticed also some different behaviour with log4j module in 1.8 -
should it be configured differently?


for completeness trimmed sample code I am trying to run:


System.setProperty("IGNITE_QUIET", "false");

Ignition.setClientMode(true);

TcpDiscoverySpi spi = new TcpDiscoverySpi();

TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();


// Set initial IP addresses.

// Note that you can optionally specify a port or a port range.

ipFinder.setAddresses(Arrays.asList("localhost:48500")); // this is
pointing to one of the cluster nodes via the ssh tunnel, works in 1.7,
list is returned, not in 1.8 vs 1.7

spi.setIpFinder(ipFinder);

spi.setLocalAddress("localhost");

Map resolverAddresses = new HashMap();

for(int i=0; i< 55 ;i++){

 String orig = String.format("192.168.168.%d:47100", 1+i); / I would
expect this IPs returned
 String dest = String.format("localhost:%03d", 47101 + i);

resolverAddresses.put(orig, dest);

}

BasicAddressResolver basicResolver = null;

try {

basicResolver = new BasicAddressResolver(resolverAddresses);

} catch (UnknownHostException e) {

// TODO Auto-generated catch block

e.printStackTrace();

}



spi.setAddressResolver(basicResolver);

IgniteConfiguration cfg = new IgniteConfiguration();



// Override default discovery SPI.

cfg.setDiscoverySpi(spi);
cfg.setAddressResolver(basicResolver); // it does not change anything
(spi, commSpi have the resolve set too).

cfg.setFailureDetectionTimeout(15000);

// Explicitly configure TCP communication SPI by changing local port number for

// the nodes from the first cluster.

TcpCommunicationSpi commSpi=new TcpCommunicationSpi();


commSpi.setLocalPort(47099); //exposed via reverse channel -R:47099:...

commSpi.setLocalPortRange(1);

commSpi.setLocalAddress("localhost");

commSpi.setAddressResolver(basicResolver);

commSpi.setSharedMemoryPort(-1);

// Overriding communication SPI.

cfg.setCommunicationSpi(commSpi);

// logging

cfg.setGridLogger(new Log4JLogger("config/ignite-log4j.xml")); //has
effect in 1.7 but not in 1.8


// Start Ignite node.

Ignite ignite = Ignition.start(cfg);


CacheConfiguration cacheCfg = new CacheConfiguration("run_cache");

cacheCfg.setCacheMode(CacheMode.PARTITIONED);

cacheCfg.setBackups(0);


Any hints?

Cheers

On Tue, Oct 4, 2016 at 1:15 AM, vkulichenko [via Apache Ignite Users]
 wrote:
> Hi Krzysztof,
>
> This issue was fixed some time ago, so in 1.8 Ignite will publish public
> addresses in the IP finder. You can with the nightly build [1] in the
> meantime.
>
> [1] https://cwiki.apache.org/confluence/display/IGNITE/Nightly+Builds
>
> -Val
>
> 
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-ignite-users.70518.x6.nabble.com/Ignite-Cluster-Communication-with-SSH-Tunnels-tp273p8066.html
> To unsubscribe from Ignite Cluster Communication with SSH Tunnels, click
> here.
> NAML




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Cluster-Communication-with-SSH-Tunnels-tp273p8081.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Couchbase as persistent store

2016-10-04 Thread kvipin 
Val,

I'm not getting any error or exception that's what makes it difficult to
figure out whats going on. But I do see Apache Ignite opening couchbase
bucket successfully, which is equivalent to connection establishment. But
after that no activity server side and client side every thing seems to be
working fine.

Following is the output from Server node:

$ ./nodemgr -c config/test-tool-server.xml
log4j:WARN No appenders could be found for logger
(org.springframework.core.env.StandardEnvironment).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
more info.
[07:04:36]__   
[07:04:36]   /  _/ ___/ |/ /  _/_  __/ __/ 
[07:04:36]  _/ // (7 7// /  / / / _/   
[07:04:36] /___/\___/_/|_/___/ /_/ /___/  
[07:04:36] 
[07:04:36] ver. 1.6.0#20160518-sha1:0b22c45b
[07:04:36] 2016 Copyright(C) Apache Software Foundation
[07:04:36] 
[07:04:36] Ignite documentation: http://ignite.apache.org
[07:04:36] 
[07:04:36] Quiet mode.
[07:04:36]   ^-- Logging to file
'/home/ignite1/project/apacheIgnite/apache-ignite-fabric-1.6.0-bin/work/log/ignite-a5293357.0.log'
[07:04:36]   ^-- To see **FULL** console log here add -DIGNITE_QUIET=false
or "-v" to ignite.{sh|bat}
[07:04:36] 
[07:04:36] OS: Linux 3.10.0-327.28.2.el7.x86_64 amd64
[07:04:36] VM information: Java(TM) SE Runtime Environment 1.8.0_92-b14
Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.92-b14
[07:04:41] Configured plugins:
[07:04:41]   ^-- None
[07:04:41] 
[07:04:42] Security status [authentication=off, tls/ssl=off]
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/home/ignite1/project/apacheIgnite/apache-ignite-fabric-1.6.0-bin/libs/slf4j-simple-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/home/ignite1/project/apacheIgnite/apache-ignite-fabric-1.6.0-bin/libs/ignite-rest-http/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory]
[main] INFO org.eclipse.jetty.util.log - Logging initialized @6767ms
[main] INFO org.eclipse.jetty.server.Server - jetty-9.2.11.v20150529
[main] INFO org.eclipse.jetty.server.ServerConnector - Started
ServerConnector@24d09c1{HTTP/1.1}{0.0.0.0:8080}
[main] INFO org.eclipse.jetty.server.Server - Started @6818ms
[07:04:44] Performance suggestions for grid  (fix if possible)
[07:04:44] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[07:04:44]   ^-- Disable peer class loading (set 'peerClassLoadingEnabled'
to false)
[07:04:44]   ^-- Enable ATOMIC mode if not using transactions (set
'atomicityMode' to ATOMIC)
[07:04:44]   ^-- Disable fully synchronous writes (set
'writeSynchronizationMode' to PRIMARY_SYNC or FULL_ASYNC)
[07:04:44] 
[07:04:44] To start Console Management & Monitoring run
ignitevisorcmd.{sh|bat}
[07:04:44] 
[07:04:44] Ignite node started OK (id=a5293357)
[07:04:44] Topology snapshot [ver=1, servers=1, clients=0, CPUs=32,
heap=0.5GB]
[07:04:55] New version is available at ignite.apache.org: 1.7.0
[07:05:46] Topology snapshot [ver=2, servers=1, clients=1, CPUs=32,
heap=1.0GB]
[flusher-0-#148%null%] INFO com.couchbase.client.core.CouchbaseCore -
CouchbaseEnvironment: {sslEnabled=false, sslKeystoreFile='null',
sslKeystorePassword=false, sslKeystore=null, bootstrapHttpEnabled=true,
bootstrapCarrierEnabled=true, bootstrapHttpDirectPort=8091,
bootstrapHttpSslPort=18091, bootstrapCarrierDirectPort=11210,
bootstrapCarrierSslPort=11207, ioPoolSize=32, computationPoolSize=32,
responseBufferSize=16384, requestBufferSize=16384, kvServiceEndpoints=1,
viewServiceEndpoints=1, queryServiceEndpoints=1, searchServiceEndpoints=1,
ioPool=NioEventLoopGroup, coreScheduler=CoreScheduler,
eventBus=DefaultEventBus, packageNameAndVersion=couchbase-java-client/2.3.3
(git: 2.3.3, core: 1.3.3), dcpEnabled=false, retryStrategy=BestEffort,
maxRequestLifetime=75000, retryDelay=ExponentialDelay{growBy 1.0
MICROSECONDS, powers of 2; lower=100, upper=10},
reconnectDelay=ExponentialDelay{growBy 1.0 MILLISECONDS, powers of 2;
lower=32, upper=4096}, observeIntervalDelay=ExponentialDelay{growBy 1.0
MICROSECONDS, powers of 2; lower=10, upper=10}, keepAliveInterval=3,
autoreleaseAfter=2000, bufferPoolingEnabled=true, tcpNodelayEnabled=true,
mutationTokensEnabled=false, socketConnectTimeout=1000,
dcpConnectionBufferSize=20971520, dcpConnectionBufferAckThreshold=0.2,
dcpConnectionName=dcp/core-io, callbacksOnIoPool=false,
disconnectTimeout=25000,
requestBufferWaitStrategy=com.couchbase.client.core.env.DefaultCoreEnvironment$2@6ec6ddb4,
queryTimeout=75000, viewTimeout=75000, kvTimeout=2500, connectTimeout=5000,
dnsSrvEnabled=false}
[cb-io-1-1] INFO com.couchbase.client.core.node.Node - Connected to Node
localhost
[cb-computations-9] INFO
com.couchbase.client.core.config.ConfigurationProvider -

Re: spark SQL thriftserver over ignite and cassandra

2016-10-04 Thread vincent gromakowski 
Hi,
I know that Ignite has SQL support but:
- ODBC driver doesn't seem to provide HTTP(S) support, which is easier to
integrate on corporate networks with rules, firewalls, proxies
- The SQL engine doesn't seem to scale like Spark SQL would. For instance,
Spark won't generate OOM is dataset (source or result) doesn't fit in
memory. From Ignite side, it's not clear...
- Spark thrift can manage multi tenancy: different users can connect to the
same SQL engine and share cache. In Ignite it's one cache per user, so a
big waste of RAM.

What I want to achieve is :
- use Cassandra for data store as it provides idempotence (HDFS/hive
doesn't), resulting in exactly once semantic without any duplicates.
- use Spark SQL thriftserver in multi tenancy for large scale adhoc
analytics queries (> TB) from an ODBC driver through HTTP(S)
- accelerate Cassandra reads when the data modeling of the Cassandra table
doesn't fit the queries. Queries would be OLAP style: target multiple C*
partitions, groupby or filters on lots of dimensions that aren't
necessarely in the C* table key.

Thanks for your advises


2016-10-04 6:51 GMT+02:00 Jörn Franke :

> I am not sure that this will be performant. What do you want to achieve
> here? Fast lookups? Then the Cassandra Ignite store might be the right
> solution. If you want to do more analytic style of queries then you can put
> the data on HDFS/Hive and use the Ignite HDFS cache to cache certain
> partitions/tables in Hive in-memory. If you want to go to iterative machine
> learning algorithms you can go for Spark on top of this. You can use then
> also Ignite cache for Spark RDDs.
>
> On 4 Oct 2016, at 02:24, Alexey Kuznetsov  wrote:
>
> Hi, Vincent!
>
> Ignite also has SQL support (also scalable), I think it will be much
> faster to query directly from Ignite than query from Spark.
> Also please mind, that before executing queries you should load all needed
> data to cache.
> To load data from Cassandra to Ignite you may use Cassandra store [1].
>
> [1] https://apacheignite.readme.io/docs/ignite-with-apache-cassandra
>
> On Tue, Oct 4, 2016 at 4:19 AM, vincent gromakowski <
> vincent.gromakow...@gmail.com> wrote:
>
>> Hi,
>> I am evaluating the possibility to use Spark SQL (and its scalability)
>> over an Ignite cache with Cassandra persistent store to increase read
>> workloads like OLAP style analytics.
>> Is there any way to configure Spark thriftserver to load an external
>> table in Ignite like we can do in Cassandra ?
>> Here is an example of config for spark backed by cassandra
>>
>> CREATE EXTERNAL TABLE MyHiveTable
>> ( id int, data string )
>> STORED BY 'org.apache.hadoop.hive.cassandra.cql.CqlStorageHandler'
>>
>> TBLPROPERTIES ("cassandra.host" = "x.x.x.x", "cassandra.ks.name"
>> = "test" ,
>>   "cassandra.cf.name" = "mytable" ,
>>   "cassandra.ks.repfactor" = "1" ,
>>   "cassandra.ks.strategy" =
>> "org.apache.cassandra.locator.SimpleStrategy" );
>>
>>
>
>
> --
> Alexey Kuznetsov
>
>

Re: Accessing igfs file through java

2016-10-04 Thread Jörn Franke 
You need to configure the igfs in the HDFS configuration file. Then you use the 
standard APIs to access HDFS files and it will go automatically through the 
cache.

> On 4 Oct 2016, at 07:35, Sateesh Karuturi  wrote:
> 
> Hello experts, 
> I am new to the Apache Ignite , and i need to access igfs file through java.
> Till now what i tried is :
> 
> import java.io.InputStream;
> 
> import org.apache.ignite.Ignite;
> import org.apache.ignite.IgniteFileSystem;
> import org.apache.ignite.Ignition;
> import org.apache.ignite.igfs.IgfsPath;
> 
> public class Ignite_Spark {
> 
>   public static void main(String[] args) throws Exception {
>   Ignite ignite = 
> Ignition.start("/usr/local/hadoop/apache-ignite-hadoop-1.6.0-bin/config/default-config.xml");
>   Ignite ignites = Ignition.ignite();
> 
>   IgniteFileSystem fs = ignites.fileSystem("igfs");
> 
>   IgfsPath workDir = new 
> IgfsPath("igfs://localhost:8020/ignite/fs");
>   fs.mkdirs(workDir);
> 
> 
>   IgfsPath filePath = new IgfsPath(workDir, "ttypes.csv");
> 
>   try (InputStream in = fs.open(filePath)) {
>   in.read();
>   }
>   }
> 
> }
> 
> 
> But getting  java.lang.IllegalArgumentException: Path should be absolute: 
> igfs://localhost:8020/ignite/fs exception.
> anyone please help me out.
> Thanks .
>