[jira] [Created] (IGNITE-2741) Ignite as Spring Session data store

[Roman Shtykh (JIRA)]

Roman Shtykh created IGNITE-2741:


 Summary: Ignite as Spring Session data store
 Key: IGNITE-2741
 URL: https://issues.apache.org/jira/browse/IGNITE-2741
 Project: Ignite
  Issue Type: New Feature
Reporter: Roman Shtykh


On Spring Session: 
http://docs.spring.io/spring-session/docs/current/reference/html5/#
How to contribute the integration: 
http://docs.spring.io/spring-session/docs/current/reference/html5/#community-contributing

_From 1.1.0 Gemfire and Hazelcast are supported and can be used for reference, 
if needed._



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-2705) .NET: Add a test to verify ToolsVersion in project and solution files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-2705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175111#comment-15175111
 ] 

ASF GitHub Bot commented on IGNITE-2705:


Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/529


> .NET: Add a test to verify ToolsVersion in project and solution files
> -
>
> Key: IGNITE-2705
> URL: https://issues.apache.org/jira/browse/IGNITE-2705
> Project: Ignite
>  Issue Type: Test
>  Components: platforms
>Affects Versions: 1.1.4
>Reporter: Pavel Tupitsyn
>Assignee: Vladimir Ozerov
> Fix For: 1.6
>
>
> We maintain VS2010 support, engineers work in newer versions, which often 
> causes incorrect ToolsVersion in csproj and sln files. Add a test to scan all 
> files and verify tools version.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (IGNITE-2705) .NET: Add a test to verify ToolsVersion in project and solution files

[Vladimir Ozerov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-2705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov closed IGNITE-2705.
---

> .NET: Add a test to verify ToolsVersion in project and solution files
> -
>
> Key: IGNITE-2705
> URL: https://issues.apache.org/jira/browse/IGNITE-2705
> Project: Ignite
>  Issue Type: Test
>  Components: platforms
>Affects Versions: 1.1.4
>Reporter: Pavel Tupitsyn
>Assignee: Vladimir Ozerov
> Fix For: 1.6
>
>
> We maintain VS2010 support, engineers work in newer versions, which often 
> causes incorrect ToolsVersion in csproj and sln files. Add a test to scan all 
> files and verify tools version.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (IGNITE-2718) modules/zookeeper/target/libs directory does not have all dependencies

[Roman Shtykh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roman Shtykh closed IGNITE-2718.


Merged

> modules/zookeeper/target/libs directory does not have all dependencies
> --
>
> Key: IGNITE-2718
> URL: https://issues.apache.org/jira/browse/IGNITE-2718
> Project: Ignite
>  Issue Type: Bug
>  Components: build, ignite-zookeeper
>Reporter: Dustin Chesterman
>Assignee: Roman Shtykh
>Priority: Critical
>  Labels: community
> Fix For: 1.6
>
>
> If you specify the zookeeper ip finder in the example cache config there are 
> a host of classnotfound exceptions.  I believe these need to be populated 
> into the build target lib dir.
> {code}
> 
> 
> 
> 
> 
>  class="org.apache.ignite.spi.discovery.tcp.ipfinder.zk.TcpDiscoveryZookeeperIpFinder">
>  value="false" />
> 
> 
>  value="192.168.200.11:2181" />
> 
> 
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (IGNITE-2710) Session not unbind from current request after invoking request.getSession().invalidate()

[Roman Shtykh (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174935#comment-15174935
 ] 

Roman Shtykh edited comment on IGNITE-2710 at 3/2/16 2:59 AM:
--

[~vkulichenko] Can I ask you to review it and mute 
IgniteWebSessionSelfTestSuite$WebSessionTransactionalSelfTest.testInvalidatedSession
 (or give me permissions)?


was (Author: roman_s):
[~vkulichenko] Can I ask you to review it?

> Session not unbind from current request after invoking 
> request.getSession().invalidate()
> 
>
> Key: IGNITE-2710
> URL: https://issues.apache.org/jira/browse/IGNITE-2710
> Project: Ignite
>  Issue Type: Bug
>  Components: cache
>Affects Versions: 1.5.0.final
> Environment: java8, tomcat8
>Reporter: YuxuanWang
>Assignee: Roman Shtykh
>  Labels: community, session, user
> Fix For: 1.6
>
>
> System.out.println(request.getSession().getId());
> request.getSession().invalidate();
> System.out.println(request.getSession().getId());
> getSession() although return the same session after the invalidation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-2710) Session not unbind from current request after invoking request.getSession().invalidate()

[Roman Shtykh (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174935#comment-15174935
 ] 

Roman Shtykh commented on IGNITE-2710:
--

[~vkulichenko] Can I ask you to review it?

> Session not unbind from current request after invoking 
> request.getSession().invalidate()
> 
>
> Key: IGNITE-2710
> URL: https://issues.apache.org/jira/browse/IGNITE-2710
> Project: Ignite
>  Issue Type: Bug
>  Components: cache
>Affects Versions: 1.5.0.final
> Environment: java8, tomcat8
>Reporter: YuxuanWang
>Assignee: Roman Shtykh
>  Labels: community, session, user
> Fix For: 1.6
>
>
> System.out.println(request.getSession().getId());
> request.getSession().invalidate();
> System.out.println(request.getSession().getId());
> getSession() although return the same session after the invalidation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-255) Jvm8 warns that MaxPermSize is ignored

[Denis Magda (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174569#comment-15174569
 ] 

Denis Magda commented on IGNITE-255:


[~samaitra], the changes look good! I've merged them into the master, thanks 
for the contribution.

[~skozlov], I've checked the changes in Windows using ignite.bat and in Cygwin 
using ignite.sh.
Please double check ignite.sh in a Linux distro and close the ticket if 
everything works fine.

> Jvm8 warns that MaxPermSize is ignored
> --
>
> Key: IGNITE-255
> URL: https://issues.apache.org/jira/browse/IGNITE-255
> Project: Ignite
>  Issue Type: Bug
>  Components: general
>Affects Versions: sprint-1
>Reporter: Sergey Kozlov
>Assignee: Saikat Maitra
>
> Jaba8 doesn't support MaxPermSize anymore. Either we should replace it in 
> ignite.sh by a similar option of Java8 or set it only if ignite.sh started 
> for Java7
> {noformat}
> D:\GridGain\Ignite-SP1\ignite-fabric-1.0.0-RC1-SNAPSHOT>bin\ignite.bat
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; 
> support was removed in 8.0
> [15:50:16]__  
> [15:50:16]   /  _/ ___/ |/ /  _/_  __/ __/
> [15:50:16]  _/ // (_ /// /  / / / _/
> [15:50:16] /___/\___/_/|_/___/ /_/ /___/
> [15:50:16]
> [15:50:16] ver. 1.0.0-RC1-SNAPSHOT#20150214-sha1:4d01989c
> [15:50:16] 2015 Copyright(C) Apache Software Foundation
> [15:50:16]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-255) Jvm8 warns that MaxPermSize is ignored

[Denis Magda (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denis Magda updated IGNITE-255:
---
Assignee: Sergey Kozlov  (was: Saikat Maitra)

> Jvm8 warns that MaxPermSize is ignored
> --
>
> Key: IGNITE-255
> URL: https://issues.apache.org/jira/browse/IGNITE-255
> Project: Ignite
>  Issue Type: Bug
>  Components: general
>Affects Versions: sprint-1
>Reporter: Sergey Kozlov
>Assignee: Sergey Kozlov
>
> Jaba8 doesn't support MaxPermSize anymore. Either we should replace it in 
> ignite.sh by a similar option of Java8 or set it only if ignite.sh started 
> for Java7
> {noformat}
> D:\GridGain\Ignite-SP1\ignite-fabric-1.0.0-RC1-SNAPSHOT>bin\ignite.bat
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; 
> support was removed in 8.0
> [15:50:16]__  
> [15:50:16]   /  _/ ___/ |/ /  _/_  __/ __/
> [15:50:16]  _/ // (_ /// /  / / / _/
> [15:50:16] /___/\___/_/|_/___/ /_/ /___/
> [15:50:16]
> [15:50:16] ver. 1.0.0-RC1-SNAPSHOT#20150214-sha1:4d01989c
> [15:50:16] 2015 Copyright(C) Apache Software Foundation
> [15:50:16]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-2630) Make 'updateCntr' available through CacheInterceptor API

[Denis Magda (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denis Magda updated IGNITE-2630:

Labels: community important  (was: )

> Make 'updateCntr' available through CacheInterceptor API
> 
>
> Key: IGNITE-2630
> URL: https://issues.apache.org/jira/browse/IGNITE-2630
> Project: Ignite
>  Issue Type: New Feature
>  Components: cache
>Affects Versions: 1.5.0.final
>Reporter: Nikolay Tikhonov
>Priority: Minor
>  Labels: community, important
>
> Make {{updateCntr}} available through {{CacheInterceptor}} API. On 
> {{onAfterPut}} and {{onAfterRemove}} methods partition update counter can be 
> got by Cache.Entry.unwrap(Long.class). When invoking other methods on 
> {{CacheIntercetor}} API ({{onBeforePut}}, {{onBeforeRemove}}) the counter is 
> not available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-1232) Support distributed SQL JOIN

[Artem Shutak (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-1232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174171#comment-15174171
 ] 

Artem Shutak commented on IGNITE-1232:
--

Added IgniteCrossCachesDistributedJoinQueryTest. Test fails for many 
configurations of caches. Need to investigate.

> Support distributed SQL JOIN 
> -
>
> Key: IGNITE-1232
> URL: https://issues.apache.org/jira/browse/IGNITE-1232
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Sergi Vladykin
>Assignee: Sergi Vladykin
> Fix For: 1.6
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (IGNITE-2700) GridClosureProcessor internal closures are [de]serialized by OptimizedMarshaller even if BinaryMarshaller is configured

[Ilya Lantukh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ilya Lantukh reassigned IGNITE-2700:


Assignee: Ilya Lantukh  (was: Vladimir Ozerov)

> GridClosureProcessor internal closures are [de]serialized by 
> OptimizedMarshaller even if BinaryMarshaller is configured
> ---
>
> Key: IGNITE-2700
> URL: https://issues.apache.org/jira/browse/IGNITE-2700
> Project: Ignite
>  Issue Type: Bug
>  Components: compute
>Affects Versions: 1.5.0.final
>Reporter: Ilya Lantukh
>Assignee: Ilya Lantukh
>  Labels: community
> Fix For: 1.6
>
>
> Usage of OptimizedMarshaller is forced because:
> a. Closures implement Externalizable.
> b. classnames.properties file contains closure class names.
> Need to implement new versions of closures (C1, C1MLA, C2, C2MLA, C4) that 
> can be processed by BinaryMarshaller.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-2740) .NET: IGNITE_HOME resolution restricts custom deployment

[Pavel Tupitsyn (JIRA)]

Pavel Tupitsyn created IGNITE-2740:
--

 Summary: .NET: IGNITE_HOME resolution restricts custom deployment
 Key: IGNITE-2740
 URL: https://issues.apache.org/jira/browse/IGNITE-2740
 Project: Ignite
  Issue Type: Bug
  Components: platforms
Affects Versions: 1.1.4
Reporter: Pavel Tupitsyn
Assignee: Pavel Tupitsyn
Priority: Blocker
 Fix For: 1.6


Need to make sure that users may deploy Ignite JARs with their applications in 
any way they want.

Currently we have quite restrictive checks in IgniteHome.IsIgniteHome method.

Documentation should also be updated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-2729) .NET: Add Troubleshooting section to the documentation

[Pavel Tupitsyn (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pavel Tupitsyn updated IGNITE-2729:
---
Description: Describe common issues and error messages (like 
"ClassNotFound", IGNITE_HOME issues, x64/x86 JVM, freeze on start, Java 
exceptions, etc), and how to fix them.  (was: Describe common issues and error 
messages (like "ClassNotFound", IGNITE_HOME issues, x64/x86 JVM, Java 
exceptions, etc), and how to fix them.)

> .NET: Add Troubleshooting section to the documentation
> --
>
> Key: IGNITE-2729
> URL: https://issues.apache.org/jira/browse/IGNITE-2729
> Project: Ignite
>  Issue Type: Task
>  Components: platforms
>Affects Versions: 1.1.4
>Reporter: Pavel Tupitsyn
>Assignee: Pavel Tupitsyn
> Fix For: 1.6
>
>
> Describe common issues and error messages (like "ClassNotFound", IGNITE_HOME 
> issues, x64/x86 JVM, freeze on start, Java exceptions, etc), and how to fix 
> them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-2705) .NET: Add a test to verify ToolsVersion in project and solution files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-2705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173960#comment-15173960
 ] 

ASF GitHub Bot commented on IGNITE-2705:


GitHub user ptupitsyn opened a pull request:

https://github.com/apache/ignite/pull/529

IGNITE-2705 .NET: Add a test to verify ToolsVersion in project and solution 
files



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ptupitsyn/ignite ignite-2705

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/529.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #529


commit a541b6ca6844b7884a2dd7cb2df205149454f1de
Author: Pavel Tupitsyn 
Date:   2016-03-01T16:12:05Z

IGNITE-2705 .NET: Add a test to verify ToolsVersion in project and solution 
files




> .NET: Add a test to verify ToolsVersion in project and solution files
> -
>
> Key: IGNITE-2705
> URL: https://issues.apache.org/jira/browse/IGNITE-2705
> Project: Ignite
>  Issue Type: Test
>  Components: platforms
>Affects Versions: 1.1.4
>Reporter: Pavel Tupitsyn
>Assignee: Pavel Tupitsyn
> Fix For: 1.6
>
>
> We maintain VS2010 support, engineers work in newer versions, which often 
> causes incorrect ToolsVersion in csproj and sln files. Add a test to scan all 
> files and verify tools version.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-2729) .NET: Add Troubleshooting section to the documentation

[Pavel Tupitsyn (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pavel Tupitsyn updated IGNITE-2729:
---
Description: Describe common issues and error messages (like 
"ClassNotFound", IGNITE_HOME issues, x64/x86 JVM, Java exceptions, etc), and 
how to fix them.  (was: Describe common issues and error messages (like 
"ClassNotFound", IGNITE_HOME issues, x64/x86 JVM, etc), and how to fix them.)

> .NET: Add Troubleshooting section to the documentation
> --
>
> Key: IGNITE-2729
> URL: https://issues.apache.org/jira/browse/IGNITE-2729
> Project: Ignite
>  Issue Type: Task
>  Components: platforms
>Affects Versions: 1.1.4
>Reporter: Pavel Tupitsyn
>Assignee: Pavel Tupitsyn
> Fix For: 1.6
>
>
> Describe common issues and error messages (like "ClassNotFound", IGNITE_HOME 
> issues, x64/x86 JVM, Java exceptions, etc), and how to fix them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-2737) Ignite-Spark documentation is missing some useful informations

[Dmitriy Setrakyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-2737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dmitriy Setrakyan updated IGNITE-2737:
--
Component/s: Ignite RDD

> Ignite-Spark documentation is missing some useful informations
> --
>
> Key: IGNITE-2737
> URL: https://issues.apache.org/jira/browse/IGNITE-2737
> Project: Ignite
>  Issue Type: Bug
>  Components: Ignite RDD
>Reporter: Luca Rea
>Assignee: Ivan Veselovsky
>  Labels: community
> Fix For: 1.6
>
>
> Hi,
> in my tests I have experienced some configuration issue running spark in 
> local and yarn-client mode, so I want to share them.
> In order to let Ignite work correctly I had to customize spark-defaults.conf 
> adding to "spark.driver.extraClassPath" and "spark.executor.extraClassPath" 
> the string
> {code}
> "/opt/ignite/libs/*:/opt/ignite/libs/optional/ignite-spark/*:/opt/ignite/libs/optional/ignite-log4j/*:/opt/ignite/libs/optional/ignite-yarn/*:/opt/ignite/libs/ignite-spring/*"
> {code}
> (opt/ignite is my IGNITE_HOME) and other IGNITE_ useful variables like 
> "spark.executorEnv.IGNITE_WORK_DIR" in order to let them be loaded by 
> executors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-2736) custom Ignite Configuration (not xml file) is not used by spark executors

[Dmitriy Setrakyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dmitriy Setrakyan updated IGNITE-2736:
--
Fix Version/s: 1.6

> custom Ignite Configuration (not xml file) is not used by spark executors
> -
>
> Key: IGNITE-2736
> URL: https://issues.apache.org/jira/browse/IGNITE-2736
> Project: Ignite
>  Issue Type: Bug
>Reporter: Luca Rea
>  Labels: community
> Fix For: 1.6
>
>
> Hi,
> I have launched an Ignite Cluster inside YARN and I use spark-shell from a 
> client to attach to existing cluster cluster in client mode, connection 
> between client and cluster doesn't support multicast so I've tried to use a 
> custom config like below:
> {code}
> import org.apache.ignite.spark._
> import org.apache.ignite.configuration._
> import org.apache.ignite._
> import org.apache.ignite.spi.discovery.tcp._
> val spi = new TcpDiscoverySpi();
> import org.apache.ignite.spi.discovery.tcp.ipfinder.vm._
> val ipFinder = new TcpDiscoveryVmIpFinder();
> import java.util.Arrays
> ipFinder.setAddresses(Arrays.asList("172.16.24.48:47500", 
> "172.16.24.49:47500", "172.16.24.50:47500", "172.16.24.51:47500", 
> "172.16.24.52:47500", "172.16.24.53:47500"));
> spi.setIpFinder(ipFinder);
> val cfg = new IgniteConfiguration() with Serializable;
> cfg.setGridName("ignite-cluster");
> cfg.setDiscoverySpi(spi);
> val cacheCfg = new CacheConfiguration("myCache");
> import org.apache.ignite.cache._
> cacheCfg.setCacheMode(CacheMode.PARTITIONED);
> cacheCfg.setBackups(1);
> cfg.setCacheConfiguration(cacheCfg);
> class cfg extends Serializable;
> val ic = new IgniteContext[Integer, Integer](sc, () => cfg )
> val sharedRdd = ic.fromCache("example")
> val x = sqlContext.sparkContext.parallelize(1 to 1, 10).map(i => (new 
> Integer(i), new Integer(i)))
> sharedRdd.savePairs(x)
> {code}
> when I run the last command it freeze waiting to connect to the cluster, in 
> fact it seems that in this way the spark execurors don't use the above 
> configuration nor load the file default-config.xml but use some hardocoded 
> configuration with only multicast enabled.
> The workaround is to use a acustom xml configuration file and copy it into 
> the config ignite path of all all spark nodes the run:
> {code}
> import org.apache.ignite.spark._
> val ic = new IgniteContext[Integer, Integer](sc, "config/custom-config.xml")
> {code}
> custom-config.xml:
> {code}
> http://www.springframework.org/schema/beans;
>xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>xsi:schemaLocation="
>http://www.springframework.org/schema/beans
>http://www.springframework.org/schema/beans/spring-beans.xsd;>
> 
> 
>   
>   
> 
>   
>  class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
>   
> 
>   172.16.24.48:47500
>   172.16.24.49:47500
>   172.16.24.50:47500
>   172.16.24.51:47500
>   172.16.24.52:47500
>   172.16.24.53:47500
> 
>   
> 
>   
> 
>   
> 
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-2738) Ignite-YARN custom queue name

[Dmitriy Setrakyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dmitriy Setrakyan updated IGNITE-2738:
--
Assignee: Ivan Veselovsky

> Ignite-YARN custom queue name
> -
>
> Key: IGNITE-2738
> URL: https://issues.apache.org/jira/browse/IGNITE-2738
> Project: Ignite
>  Issue Type: Bug
>Reporter: Luca Rea
>Assignee: Ivan Veselovsky
>  Labels: community
> Fix For: 1.6
>
>
> Hi,
> Actually is not possible to pass a custom queue name when launching Ignite 
> YARN, so all Ignite YARN containers always run into the "default" queue (that 
> usually is configured to have limited resources available).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-2736) custom Ignite Configuration (not xml file) is not used by spark executors

[Dmitriy Setrakyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dmitriy Setrakyan updated IGNITE-2736:
--
Component/s: Ignite RDD

> custom Ignite Configuration (not xml file) is not used by spark executors
> -
>
> Key: IGNITE-2736
> URL: https://issues.apache.org/jira/browse/IGNITE-2736
> Project: Ignite
>  Issue Type: Bug
>  Components: Ignite RDD
>Reporter: Luca Rea
>Assignee: Ivan Veselovsky
>  Labels: community
> Fix For: 1.6
>
>
> Hi,
> I have launched an Ignite Cluster inside YARN and I use spark-shell from a 
> client to attach to existing cluster cluster in client mode, connection 
> between client and cluster doesn't support multicast so I've tried to use a 
> custom config like below:
> {code}
> import org.apache.ignite.spark._
> import org.apache.ignite.configuration._
> import org.apache.ignite._
> import org.apache.ignite.spi.discovery.tcp._
> val spi = new TcpDiscoverySpi();
> import org.apache.ignite.spi.discovery.tcp.ipfinder.vm._
> val ipFinder = new TcpDiscoveryVmIpFinder();
> import java.util.Arrays
> ipFinder.setAddresses(Arrays.asList("172.16.24.48:47500", 
> "172.16.24.49:47500", "172.16.24.50:47500", "172.16.24.51:47500", 
> "172.16.24.52:47500", "172.16.24.53:47500"));
> spi.setIpFinder(ipFinder);
> val cfg = new IgniteConfiguration() with Serializable;
> cfg.setGridName("ignite-cluster");
> cfg.setDiscoverySpi(spi);
> val cacheCfg = new CacheConfiguration("myCache");
> import org.apache.ignite.cache._
> cacheCfg.setCacheMode(CacheMode.PARTITIONED);
> cacheCfg.setBackups(1);
> cfg.setCacheConfiguration(cacheCfg);
> class cfg extends Serializable;
> val ic = new IgniteContext[Integer, Integer](sc, () => cfg )
> val sharedRdd = ic.fromCache("example")
> val x = sqlContext.sparkContext.parallelize(1 to 1, 10).map(i => (new 
> Integer(i), new Integer(i)))
> sharedRdd.savePairs(x)
> {code}
> when I run the last command it freeze waiting to connect to the cluster, in 
> fact it seems that in this way the spark execurors don't use the above 
> configuration nor load the file default-config.xml but use some hardocoded 
> configuration with only multicast enabled.
> The workaround is to use a acustom xml configuration file and copy it into 
> the config ignite path of all all spark nodes the run:
> {code}
> import org.apache.ignite.spark._
> val ic = new IgniteContext[Integer, Integer](sc, "config/custom-config.xml")
> {code}
> custom-config.xml:
> {code}
> http://www.springframework.org/schema/beans;
>xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>xsi:schemaLocation="
>http://www.springframework.org/schema/beans
>http://www.springframework.org/schema/beans/spring-beans.xsd;>
> 
> 
>   
>   
> 
>   
>  class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
>   
> 
>   172.16.24.48:47500
>   172.16.24.49:47500
>   172.16.24.50:47500
>   172.16.24.51:47500
>   172.16.24.52:47500
>   172.16.24.53:47500
> 
>   
> 
>   
> 
>   
> 
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-2737) Ignite-Spark documentation is missing some useful informations

[Dmitriy Setrakyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-2737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dmitriy Setrakyan updated IGNITE-2737:
--
Assignee: Ivan Veselovsky

> Ignite-Spark documentation is missing some useful informations
> --
>
> Key: IGNITE-2737
> URL: https://issues.apache.org/jira/browse/IGNITE-2737
> Project: Ignite
>  Issue Type: Bug
>Reporter: Luca Rea
>Assignee: Ivan Veselovsky
>  Labels: community
> Fix For: 1.6
>
>
> Hi,
> in my tests I have experienced some configuration issue running spark in 
> local and yarn-client mode, so I want to share them.
> In order to let Ignite work correctly I had to customize spark-defaults.conf 
> adding to "spark.driver.extraClassPath" and "spark.executor.extraClassPath" 
> the string
> {code}
> "/opt/ignite/libs/*:/opt/ignite/libs/optional/ignite-spark/*:/opt/ignite/libs/optional/ignite-log4j/*:/opt/ignite/libs/optional/ignite-yarn/*:/opt/ignite/libs/ignite-spring/*"
> {code}
> (opt/ignite is my IGNITE_HOME) and other IGNITE_ useful variables like 
> "spark.executorEnv.IGNITE_WORK_DIR" in order to let them be loaded by 
> executors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-2736) custom Ignite Configuration (not xml file) is not used by spark executors

[Dmitriy Setrakyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dmitriy Setrakyan updated IGNITE-2736:
--
Labels: community  (was: )

> custom Ignite Configuration (not xml file) is not used by spark executors
> -
>
> Key: IGNITE-2736
> URL: https://issues.apache.org/jira/browse/IGNITE-2736
> Project: Ignite
>  Issue Type: Bug
>Reporter: Luca Rea
>  Labels: community
> Fix For: 1.6
>
>
> Hi,
> I have launched an Ignite Cluster inside YARN and I use spark-shell from a 
> client to attach to existing cluster cluster in client mode, connection 
> between client and cluster doesn't support multicast so I've tried to use a 
> custom config like below:
> {code}
> import org.apache.ignite.spark._
> import org.apache.ignite.configuration._
> import org.apache.ignite._
> import org.apache.ignite.spi.discovery.tcp._
> val spi = new TcpDiscoverySpi();
> import org.apache.ignite.spi.discovery.tcp.ipfinder.vm._
> val ipFinder = new TcpDiscoveryVmIpFinder();
> import java.util.Arrays
> ipFinder.setAddresses(Arrays.asList("172.16.24.48:47500", 
> "172.16.24.49:47500", "172.16.24.50:47500", "172.16.24.51:47500", 
> "172.16.24.52:47500", "172.16.24.53:47500"));
> spi.setIpFinder(ipFinder);
> val cfg = new IgniteConfiguration() with Serializable;
> cfg.setGridName("ignite-cluster");
> cfg.setDiscoverySpi(spi);
> val cacheCfg = new CacheConfiguration("myCache");
> import org.apache.ignite.cache._
> cacheCfg.setCacheMode(CacheMode.PARTITIONED);
> cacheCfg.setBackups(1);
> cfg.setCacheConfiguration(cacheCfg);
> class cfg extends Serializable;
> val ic = new IgniteContext[Integer, Integer](sc, () => cfg )
> val sharedRdd = ic.fromCache("example")
> val x = sqlContext.sparkContext.parallelize(1 to 1, 10).map(i => (new 
> Integer(i), new Integer(i)))
> sharedRdd.savePairs(x)
> {code}
> when I run the last command it freeze waiting to connect to the cluster, in 
> fact it seems that in this way the spark execurors don't use the above 
> configuration nor load the file default-config.xml but use some hardocoded 
> configuration with only multicast enabled.
> The workaround is to use a acustom xml configuration file and copy it into 
> the config ignite path of all all spark nodes the run:
> {code}
> import org.apache.ignite.spark._
> val ic = new IgniteContext[Integer, Integer](sc, "config/custom-config.xml")
> {code}
> custom-config.xml:
> {code}
> http://www.springframework.org/schema/beans;
>xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>xsi:schemaLocation="
>http://www.springframework.org/schema/beans
>http://www.springframework.org/schema/beans/spring-beans.xsd;>
> 
> 
>   
>   
> 
>   
>  class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
>   
> 
>   172.16.24.48:47500
>   172.16.24.49:47500
>   172.16.24.50:47500
>   172.16.24.51:47500
>   172.16.24.52:47500
>   172.16.24.53:47500
> 
>   
> 
>   
> 
>   
> 
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-2737) Ignite-Spark documentation is missing some useful informations

[Dmitriy Setrakyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-2737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dmitriy Setrakyan updated IGNITE-2737:
--
Labels: community  (was: )

> Ignite-Spark documentation is missing some useful informations
> --
>
> Key: IGNITE-2737
> URL: https://issues.apache.org/jira/browse/IGNITE-2737
> Project: Ignite
>  Issue Type: Bug
>Reporter: Luca Rea
>  Labels: community
> Fix For: 1.6
>
>
> Hi,
> in my tests I have experienced some configuration issue running spark in 
> local and yarn-client mode, so I want to share them.
> In order to let Ignite work correctly I had to customize spark-defaults.conf 
> adding to "spark.driver.extraClassPath" and "spark.executor.extraClassPath" 
> the string
> {code}
> "/opt/ignite/libs/*:/opt/ignite/libs/optional/ignite-spark/*:/opt/ignite/libs/optional/ignite-log4j/*:/opt/ignite/libs/optional/ignite-yarn/*:/opt/ignite/libs/ignite-spring/*"
> {code}
> (opt/ignite is my IGNITE_HOME) and other IGNITE_ useful variables like 
> "spark.executorEnv.IGNITE_WORK_DIR" in order to let them be loaded by 
> executors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-2738) Ignite-YARN custom queue name

[Dmitriy Setrakyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dmitriy Setrakyan updated IGNITE-2738:
--
Fix Version/s: 1.6

> Ignite-YARN custom queue name
> -
>
> Key: IGNITE-2738
> URL: https://issues.apache.org/jira/browse/IGNITE-2738
> Project: Ignite
>  Issue Type: Bug
>Reporter: Luca Rea
>  Labels: community
> Fix For: 1.6
>
>
> Hi,
> Actually is not possible to pass a custom queue name when launching Ignite 
> YARN, so all Ignite YARN containers always run into the "default" queue (that 
> usually is configured to have limited resources available).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (IGNITE-2738) Ignite-YARN custom queue name

[Dmitriy Setrakyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dmitriy Setrakyan updated IGNITE-2738:
--
Labels: community  (was: )

> Ignite-YARN custom queue name
> -
>
> Key: IGNITE-2738
> URL: https://issues.apache.org/jira/browse/IGNITE-2738
> Project: Ignite
>  Issue Type: Bug
>Reporter: Luca Rea
>  Labels: community
>
> Hi,
> Actually is not possible to pass a custom queue name when launching Ignite 
> YARN, so all Ignite YARN containers always run into the "default" queue (that 
> usually is configured to have limited resources available).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-1419) .Net: Add optional "raw" flag to binary type configuration.

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173886#comment-15173886
 ] 

Pavel Tupitsyn commented on IGNITE-1419:


Vladimir, this approach is fine for classes that just hold a bunch of 
properties.
In this case, making writeable property results in a class with inconsistent 
behavior. This is just bad design.
Yes, in our current system it is very unlikely for the user to change that 
property after starting Ignite, but if you look at this class separately, it is 
broken.
And there is no reason to do this.

> .Net: Add optional "raw" flag to binary type configuration.
> ---
>
> Key: IGNITE-1419
> URL: https://issues.apache.org/jira/browse/IGNITE-1419
> Project: Ignite
>  Issue Type: Task
>  Components: platforms
>Affects Versions: 1.1.4
>Reporter: Vladimir Ozerov
>Assignee: Pavel Tupitsyn
> Fix For: 1.6
>
>
> By default we write .Net objects with field names included. But in some cases 
> it is not needed. E.g. when objects are not going to be used in queries.
> Removing field names will speed up serialization/deserialization. We can add 
> optional "raw" flag to portable type descriptor to control this behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-1630) .Net: Create LINQ adapter for queries.

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173872#comment-15173872
 ] 

Pavel Tupitsyn commented on IGNITE-1630:


All done, local mode added.

> .Net: Create LINQ adapter for queries.
> --
>
> Key: IGNITE-1630
> URL: https://issues.apache.org/jira/browse/IGNITE-1630
> Project: Ignite
>  Issue Type: Task
>  Components: platforms
>Affects Versions: ignite-1.4
>Reporter: Vladimir Ozerov
>Assignee: Pavel Tupitsyn
>Priority: Critical
> Fix For: 1.6
>
>
> SQL queries are one of the most frequently used features in data grids. And 
> .Net comes with a very nice LINQ concept. 
> * LINQ provider should come in a separate assembly
> * Make sure that assembly is included in binary distribution



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (IGNITE-640) Implement IgniteMultimap data structures

[Konstantin Margorin (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173717#comment-15173717
 ] 

Konstantin Margorin edited comment on IGNITE-640 at 3/1/16 2:20 PM:


Some thoughts about possible multimap  implementation.

According to above suggestion we shell use cache with specially constructed key:

{code}
class MultiKey {
@CacheAffinityMapped
private K key;
 
int index;
}
{code}
 
Method {{put(K key, V value)}} should created MultiKey for a new value. We 
should know new index value for MultiKey construction in the {{put}} method. 
The simplest implementation will use another {{Cache indexCache}}   to 
keep count of items with key K.
 
So, pseudo-code for put will look like

{code} 
put(K key, V value)
{
try (Transaction tx = Ignition.ignite().transactions().txStart()) { 
newIndex = indexCache.get(key) + 1;
indexCache.put(key, newIndex);
multikeyCache.put(new MultiKey(key, newIndex), value);
tx.commit()
}
}
{code}
 
The problem here is that we use:

- {{transaction}}
- one {{get}}
- two {{put}}

cache operations for one Multikey {{put}} operation. It looks like big overhead.

Dmitriy suggested another way:
{quote}
An alternative way would be to store Key to List, i. e. to store Lists 
directly in cache. Then we can use EntryProcessor to add/remove elements to the 
list. The only problem here is that we will not be able to update the existing 
lists, but rather clone the lists, so we can add/remove elements to it, but I 
still think it will be cheaper than using transactions.
{quote}


was (Author: ruskim):
Some thoughts possible multimap about implementation.

According to above suggestion we shell use cache with specially constructed key:

{code}
class MultiKey {
@CacheAffinityMapped
private K key;
 
int index;
}
{code}
 
Method {{put(K key, V value)}} should created MultiKey for a new value. We 
should know new index value for MultiKey construction in the {{put}} method. 
The simplest implementation will use another {{Cache indexCache}}   to 
keep count of items with key K.
 
So, pseudo-code for put will look like

{code} 
put(K key, V value)
{
try (Transaction tx = Ignition.ignite().transactions().txStart()) { 
newIndex = indexCache.get(key) + 1;
indexCache.put(key, newIndex);
multikeyCache.put(new MultiKey(key, newIndex), value);
tx.commit()
}
}
{code}
 
The problem here is that we use:

- {{transaction}}
- one {{get}}
- two {{put}}

cache operations for one Multikey {{put}} operation. It looks like big overhead.

Dmitriy suggested another way:
{quote}
An alternative way would be to store Key to List, i. e. to store Lists 
directly in cache. Then we can use EntryProcessor to add/remove elements to the 
list. The only problem here is that we will not be able to update the existing 
lists, but rather clone the lists, so we can add/remove elements to it, but I 
still think it will be cheaper than using transactions.
{quote}

> Implement IgniteMultimap data structures
> 
>
> Key: IGNITE-640
> URL: https://issues.apache.org/jira/browse/IGNITE-640
> Project: Ignite
>  Issue Type: Sub-task
>  Components: data structures
>Reporter: Dmitriy Setrakyan
>Assignee: Konstantin Margorin
>
> We need to add {{IgniteMultimap}} data structure in addition to other data 
> structures provided by Ignite. {{IgniteMultiMap}} should have similar API to 
> {{java.util.Map}} class in JDK, but support the semantics of multiple values 
> per key, similar to [Guava 
> Multimap|http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/collect/Multimap.html].
>  
> However, unlike in Guava, our multi-map should work with Lists, not 
> Collections. Lists should make it possible to support the following methods:
> {code}
> // Gets value at a certain index for a key.
> V get(K, index);
> // Gets all values for a collection of keys at a certain index.
> Map getAll(Collection, index);
> // Gets values for specified indexes for a key.
> List get(K, Iterable indexes);
> // Gets all values for a collection of keys at specified indexes.
> Map getAll(Collection, Iterable indexes);
> // Gets values for specified range of indexes, between min and max.
> List get(K, int min, int max);
> // Gets all values for a collection of keys for a specified index range, 
> between min and max.
> Map getAll(Collection, int min, int max);
> // Gets all values for a specific key.
> List get(K);
> // Gets all values for a collection of keys.
> Map getAll(Collection);
> // Iterate through all elements with a certain index.
> Iterator> iterate(int idx);
> // Do we need this?
> 

[jira] [Commented] (IGNITE-2734) Preloading fails during entry preloading for binary enum with OFFHEAP_VALUES cache memory mode

[Andrey Gura (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-2734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173792#comment-15173792
 ] 

Andrey Gura commented on IGNITE-2734:
-

Tests added. 

> Preloading fails during entry preloading for binary enum with OFFHEAP_VALUES 
> cache memory mode 
> ---
>
> Key: IGNITE-2734
> URL: https://issues.apache.org/jira/browse/IGNITE-2734
> Project: Ignite
>  Issue Type: Bug
>Reporter: Andrey Gura
>Assignee: Andrey Gura
> Fix For: 1.6
>
>
> Preloading fails during preloading of entry for {{BinaryEnumObjectImpl}} 
> value type and cache memory mode is {{OFFHEAP_VALUES}}. 
> {{GridCacheReplicatedPreloadOffHeapSelfTest.testDeployment}} fails on TC.
> {noformat}
> java.lang.AssertionError
> at 
> org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMemory.putOffHeap(GridUnsafeMemory.java:413)
> at 
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.value(GridCacheMapEntry.java:252)
> at 
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.update(GridCacheMapEntry.java:2916)
> at 
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.initialValue(GridCacheMapEntry.java:3258)
> at 
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionDemander.preloadEntry(GridDhtPartitionDemander.java:683)
> at 
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionDemander.handleSupplyMessage(GridDhtPartitionDemander.java:572)
> at 
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPreloader.handleSupplyMessage(GridDhtPreloader.java:408)
> at 
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$5.apply(GridCachePartitionExchangeManager.java:342)
> at 
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$5.apply(GridCachePartitionExchangeManager.java:335)
> at 
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:605)
> at 
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:303)
> at 
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$400(GridCacheIoManager.java:81)
> at 
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$OrderedMessageListener.onMessage(GridCacheIoManager.java:1108)
> at 
> org.apache.ignite.internal.managers.communication.GridIoManager$GridCommunicationMessageSet.unwind(GridIoManager.java:2239)
> at 
> org.apache.ignite.internal.managers.communication.GridIoManager.unwindMessageSet(GridIoManager.java:1004)
> at 
> org.apache.ignite.internal.managers.communication.GridIoManager.access$1800(GridIoManager.java:103)
> at 
> org.apache.ignite.internal.managers.communication.GridIoManager$6.run(GridIoManager.java:973)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-2739) .NET: AffinityKey support

[Pavel Tupitsyn (JIRA)]

Pavel Tupitsyn created IGNITE-2739:
--

 Summary: .NET: AffinityKey support
 Key: IGNITE-2739
 URL: https://issues.apache.org/jira/browse/IGNITE-2739
 Project: Ignite
  Issue Type: Task
  Components: platforms
Affects Versions: 1.1.4
Reporter: Pavel Tupitsyn
Assignee: Pavel Tupitsyn
Priority: Critical
 Fix For: 1.6


Add AffinityKey to allow custom affinity mapping.
This is crucial for SQL joins to work properly.
See org.apache.ignite.cache.affinity.AffinityKey in Java.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-640) Implement IgniteMultimap data structures

[Konstantin Margorin (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173717#comment-15173717
 ] 

Konstantin Margorin commented on IGNITE-640:


Some thoughts possible multimap about implementation.

According to above suggestion we shell use cache with specially constructed key:

{code}
class MultiKey {
@CacheAffinityMapped
private K key;
 
int index;
}
{code}
 
Method {{put(K key, V value)}} should created MultiKey for a new value. We 
should know new index value for MultiKey construction in the {{put}} method. 
The simplest implementation will use another {{Cache indexCache}}   to 
keep count of items with key K.
 
So, pseudo-code for put will look like

{code} 
put(K key, V value)
{
try (Transaction tx = Ignition.ignite().transactions().txStart()) { 
newIndex = indexCache.get(key) + 1;
indexCache.put(key, newIndex);
multikeyCache.put(new MultiKey(key, newIndex), value);
tx.commit()
}
}
{code}
 
The problem here is that we use:

- {{transaction}}
- one {{get}}
- two {{put}}

cache operations for one Multikey {{put}} operation. It looks like big overhead.

Dmitriy suggested another way:
{quote}
An alternative way would be to store Key to List, i. e. to store Lists 
directly in cache. Then we can use EntryProcessor to add/remove elements to the 
list. The only problem here is that we will not be able to update the existing 
lists, but rather clone the lists, so we can add/remove elements to it, but I 
still think it will be cheaper than using transactions.
{quote}

> Implement IgniteMultimap data structures
> 
>
> Key: IGNITE-640
> URL: https://issues.apache.org/jira/browse/IGNITE-640
> Project: Ignite
>  Issue Type: Sub-task
>  Components: data structures
>Reporter: Dmitriy Setrakyan
>Assignee: Konstantin Margorin
>
> We need to add {{IgniteMultimap}} data structure in addition to other data 
> structures provided by Ignite. {{IgniteMultiMap}} should have similar API to 
> {{java.util.Map}} class in JDK, but support the semantics of multiple values 
> per key, similar to [Guava 
> Multimap|http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/collect/Multimap.html].
>  
> However, unlike in Guava, our multi-map should work with Lists, not 
> Collections. Lists should make it possible to support the following methods:
> {code}
> // Gets value at a certain index for a key.
> V get(K, index);
> // Gets all values for a collection of keys at a certain index.
> Map getAll(Collection, index);
> // Gets values for specified indexes for a key.
> List get(K, Iterable indexes);
> // Gets all values for a collection of keys at specified indexes.
> Map getAll(Collection, Iterable indexes);
> // Gets values for specified range of indexes, between min and max.
> List get(K, int min, int max);
> // Gets all values for a collection of keys for a specified index range, 
> between min and max.
> Map getAll(Collection, int min, int max);
> // Gets all values for a specific key.
> List get(K);
> // Gets all values for a collection of keys.
> Map getAll(Collection);
> // Iterate through all elements with a certain index.
> Iterator> iterate(int idx);
> // Do we need this?
> Collection get(K, IgniteBiPredicate)
> {code}
> Multimap should also support colocated and non-colocated modes, similar to 
> [IgniteQueue|https://github.com/apache/incubator-ignite/blob/master/modules/core/src/main/java/org/apache/ignite/IgniteQueue.java]
>  and its implementation, 
> [GridAtomicCacheQueueImpl|https://github.com/apache/incubator-ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/datastructures/GridAtomicCacheQueueImpl.java].
> h2. Design Details
> The most natural way to implement such map, would be to store every value 
> under a separate key in an Ignite cache. For example, let's say that we have 
> a key {{K}} with multiple values: {{V0, V1, V2, ...}}. Then the cache should 
> end up with the following values {{K0, V0}}, {{K1, V1}}, {{K2, V2}}, etc. 
> This means that we need to wrap user key into our own, internal key, which 
> will also have {{index}} field. 
> Also note that we need to collocate all the values for the same key on the 
> same node, which means that we need to define user key K as the affinity key, 
> like so:
> {code}
> class MultiKey {
> @CacheAffinityMapped
> private K key;
> int index;
> }
> {code}
> Look ups of values at specific indexes becomes very simple. Just attach a 
> specific index to a key and do a cache lookup. Look ups for all values for a 
> key should work as following:
> {code}
> MultiKey key;
> V v = null;
> int index = 0;
> List res = new LinkedList<>();

[jira] [Commented] (IGNITE-1630) .Net: Create LINQ adapter for queries.

[Vladimir Ozerov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173660#comment-15173660
 ] 

Vladimir Ozerov commented on IGNITE-1630:
-

One more thing: we need to be able to execute both distributed and *local* 
queries.

> .Net: Create LINQ adapter for queries.
> --
>
> Key: IGNITE-1630
> URL: https://issues.apache.org/jira/browse/IGNITE-1630
> Project: Ignite
>  Issue Type: Task
>  Components: platforms
>Affects Versions: ignite-1.4
>Reporter: Vladimir Ozerov
>Assignee: Pavel Tupitsyn
>Priority: Critical
> Fix For: 1.6
>
>
> SQL queries are one of the most frequently used features in data grids. And 
> .Net comes with a very nice LINQ concept. 
> * LINQ provider should come in a separate assembly
> * Make sure that assembly is included in binary distribution



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-1630) .Net: Create LINQ adapter for queries.

[Vladimir Ozerov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173659#comment-15173659
 ] 

Vladimir Ozerov commented on IGNITE-1630:
-

Pavel, my thoughts on this:
3) Looks like we do not have real need for this for now. Let's remove and add 
later if needed.
4) Got it. I would move it to private interface then.
5) This way some LINQ internal mechanics will be exposed to a class in Core 
module. Let's handle this as a special case in LINQ module instead.
6) Should be addressed in a separate ticket.
9) Let's return just cache name then.
10) This string could be used in SQL fields query. So let's rename it to 
something like "GetSqlFieldsQueryString()". Or better - return SqlFieldsQuery 
with both query string and arguments set.

> .Net: Create LINQ adapter for queries.
> --
>
> Key: IGNITE-1630
> URL: https://issues.apache.org/jira/browse/IGNITE-1630
> Project: Ignite
>  Issue Type: Task
>  Components: platforms
>Affects Versions: ignite-1.4
>Reporter: Vladimir Ozerov
>Assignee: Pavel Tupitsyn
>Priority: Critical
> Fix For: 1.6
>
>
> SQL queries are one of the most frequently used features in data grids. And 
> .Net comes with a very nice LINQ concept. 
> * LINQ provider should come in a separate assembly
> * Make sure that assembly is included in binary distribution



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-2734) Preloading fails during entry preloading for binary enum with OFFHEAP_VALUES cache memory mode

[Semen Boikov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-2734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173653#comment-15173653
 ] 

Semen Boikov commented on IGNITE-2734:
--

Changes look good, but I think we need add more tests, especially to check that 
'className' is properly deserialized.

> Preloading fails during entry preloading for binary enum with OFFHEAP_VALUES 
> cache memory mode 
> ---
>
> Key: IGNITE-2734
> URL: https://issues.apache.org/jira/browse/IGNITE-2734
> Project: Ignite
>  Issue Type: Bug
>Reporter: Andrey Gura
>Assignee: Andrey Gura
> Fix For: 1.6
>
>
> Preloading fails during preloading of entry for {{BinaryEnumObjectImpl}} 
> value type and cache memory mode is {{OFFHEAP_VALUES}}. 
> {{GridCacheReplicatedPreloadOffHeapSelfTest.testDeployment}} fails on TC.
> {noformat}
> java.lang.AssertionError
> at 
> org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMemory.putOffHeap(GridUnsafeMemory.java:413)
> at 
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.value(GridCacheMapEntry.java:252)
> at 
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.update(GridCacheMapEntry.java:2916)
> at 
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.initialValue(GridCacheMapEntry.java:3258)
> at 
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionDemander.preloadEntry(GridDhtPartitionDemander.java:683)
> at 
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionDemander.handleSupplyMessage(GridDhtPartitionDemander.java:572)
> at 
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPreloader.handleSupplyMessage(GridDhtPreloader.java:408)
> at 
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$5.apply(GridCachePartitionExchangeManager.java:342)
> at 
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$5.apply(GridCachePartitionExchangeManager.java:335)
> at 
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:605)
> at 
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:303)
> at 
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$400(GridCacheIoManager.java:81)
> at 
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$OrderedMessageListener.onMessage(GridCacheIoManager.java:1108)
> at 
> org.apache.ignite.internal.managers.communication.GridIoManager$GridCommunicationMessageSet.unwind(GridIoManager.java:2239)
> at 
> org.apache.ignite.internal.managers.communication.GridIoManager.unwindMessageSet(GridIoManager.java:1004)
> at 
> org.apache.ignite.internal.managers.communication.GridIoManager.access$1800(GridIoManager.java:103)
> at 
> org.apache.ignite.internal.managers.communication.GridIoManager$6.run(GridIoManager.java:973)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (IGNITE-1630) .Net: Create LINQ adapter for queries.

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173607#comment-15173607
 ] 

Pavel Tupitsyn edited comment on IGNITE-1630 at 3/1/16 11:38 AM:
-

1) Done
2) Done (except fields queries, where this is not possible)
3) Allows to specify Java type name to be used in SQL index, if the user wants 
to override default mapping
4) Improves performance of the fields query. Instead allocating a List 
and then unboxing the values from it, we can read fields directly as specific 
types.
LINQ provider uses this method when doing fields queries. I'd like to avoid 
adding another InternalsVisibleTo for LINQ project, so this has to be on public 
API. 

And by itself, this method is not only faster than old one, but also more 
usable.
{code}cache.QueryFields((r,c)=>new MyClass(r.ReadInt(), r.ReadString())){code} 
is much better than working with untyped list.

5) Ignite SQL has reserved "_key" and "_val" field names to query cache entry 
key or value as a whole. LINQ provider looks at [QuerySqlField] attributes to 
generate field names. 
6) This is just a placeholder. It is much better than "Console.WriteLine" 
scattered all over the codebase that we had before. There are separate tasks 
for logging, I just did not want to add more WriteLine calls.
7) AsQueryable is an existing extension method in System.Linq.Queryable. Our 
method should stand out, because it implies a totally different mechanism 
underneath.
8) Yes, we do, it provides a way to eliminate LINQ translation overhead. The 
design is the same as in Entity Framework and will be familiar to the users. 
https://msdn.microsoft.com/en-us/library/system.data.linq.compiledquery(v=vs.110).aspx
9) Yes, but ICache is generic. We can't return original cache instance in a 
non-generic interface. 
10) This name is from Entity Framework: 
https://msdn.microsoft.com/en-us/library/system.data.objects.objectquery.totracestring(v=vs.110).aspx
11) Any query that originates from our ToCacheQueryable can be cast to this 
interface to obtain resulting SQL and other things. Again, same thing as in EF. 
LINQ is just a set of extension methods, they all return IQueryable, so there 
is no way to operate on our own interface.
12) This interface is for query provider implementers, not for users. Users 
operate solely on IQueryable.


was (Author: ptupitsyn):
1) Done
2) Done (except fields queries, where this is not possible)
3) Allows to specify Java type name to be used in SQL index, if the user wants 
to override default mapping
4) Improves performance of the fields query. Instead allocating a List 
and then unboxing the values from it, we can read fields directly as specific 
types.
LINQ provider uses this method when doing fields queries. I'd like to avoid 
adding another InternalsVisibleTo for LINQ project, so this has to be on public 
API. 

And by itself, this method is not only faster than old one, but also more 
usable.
{code}cache.QueryFields((r,c)=>new MyClass(r.ReadInt(), r.ReadString())){code} 
is much better than working with untyped list.

5) Ignite SQL has reserved "_key" and "_val" field names to query cache entry 
key or value as a whole. LINQ provider looks at [QuerySqlField] attributes to 
generate field names. 
6) This is just a placeholder. It is much better than "Console.WriteLine" 
scattered all over the codebase that we had before. There are separate tasks 
for logging, I just did not want to add more WriteLine calls.
7) AsQueryable is an existing extension method in System.Linq.Queryable. Our 
method should stand out, because it implies a totally different mechanism 
underneath.
8) Yes, we do, it provides a way to eliminate LINQ translation overhead. The 
design is the same as in Entity Framework and will be familiar to the users. 
https://msdn.microsoft.com/en-us/library/system.data.linq.compiledquery(v=vs.110).aspx
9) Yes, but ICache is generic. We can't return original cache instance in a 
non-generic interface. 
10) This name is from Entity Framework: 
https://msdn.microsoft.com/en-us/library/system.data.objects.objectquery.totracestring(v=vs.110).aspx
11) Any query that originates from our ToCacheQueryable can be cast to this 
interface to obtain resulting SQL and other things. Again, same thing as in EF. 
LINQ is just a set of extension methods, they all return IQueryable, so there 
is no way to operate on our own interface.

> .Net: Create LINQ adapter for queries.
> --
>
> Key: IGNITE-1630
> URL: https://issues.apache.org/jira/browse/IGNITE-1630
> Project: Ignite
>  Issue Type: Task
>  Components: platforms
>Affects Versions: ignite-1.4
>Reporter: Vladimir Ozerov
>Assignee: Pavel Tupitsyn
>Priority: Critical
> Fix For: 1.6
>
>
> SQL queries are one of the most 

[jira] [Comment Edited] (IGNITE-1630) .Net: Create LINQ adapter for queries.

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173607#comment-15173607
 ] 

Pavel Tupitsyn edited comment on IGNITE-1630 at 3/1/16 11:34 AM:
-

1) Done
2) Done (except fields queries, where this is not possible)
3) Allows to specify Java type name to be used in SQL index, if the user wants 
to override default mapping
4) Improves performance of the fields query. Instead allocating a List 
and then unboxing the values from it, we can read fields directly as specific 
types.
LINQ provider uses this method when doing fields queries. I'd like to avoid 
adding another InternalsVisibleTo for LINQ project, so this has to be on public 
API. 

And by itself, this method is not only faster than old one, but also more 
usable.
{code}cache.QueryFields((r,c)=>new MyClass(r.ReadInt(), r.ReadString())){code} 
is much better than working with untyped list.

5) Ignite SQL has reserved "_key" and "_val" field names to query cache entry 
key or value as a whole. LINQ provider looks at [QuerySqlField] attributes to 
generate field names. 
6) This is just a placeholder. It is much better than "Console.WriteLine" 
scattered all over the codebase that we had before. There are separate tasks 
for logging, I just did not want to add more WriteLine calls.
7) AsQueryable is an existing extension method in System.Linq.Queryable. Our 
method should stand out, because it implies a totally different mechanism 
underneath.
8) Yes, we do, it provides a way to eliminate LINQ translation overhead. The 
design is the same as in Entity Framework and will be familiar to the users. 
https://msdn.microsoft.com/en-us/library/system.data.linq.compiledquery(v=vs.110).aspx
9) Yes, but ICache is generic. We can't return original cache instance in a 
non-generic interface. 
10) This name is from Entity Framework: 
https://msdn.microsoft.com/en-us/library/system.data.objects.objectquery.totracestring(v=vs.110).aspx
11) Any query that originates from our ToCacheQueryable can be cast to this 
interface to obtain resulting SQL and other things. Again, same thing as in EF. 
LINQ is just a set of extension methods, they all return IQueryable, so there 
is no way to operate on our own interface.


was (Author: ptupitsyn):
1) Done
2) Done (except fields queries, where this is not possible)
3) Allows to specify Java type name to be used in SQL index, if the user wants 
to override default mapping
4) Improves performance of the fields query. Instead allocating a List 
and then unboxing the values from it, we can read fields directly as specific 
types.
LINQ provider uses this method when doing fields queries. I'd like to avoid 
adding another InternalsVisibleTo for LINQ project, so this has to be on public 
API. 

And by itself, this method is not only faster than old one, but also more 
usable.
{code}cache.QueryFields((r,c)=>new MyClass(r.ReadInt(), r.ReadString())){code} 
is much better than working with untyped list.

5) Ignite SQL has reserved "_key" and "_val" field names to query cache entry 
key or value as a whole. LINQ provider looks at [QuerySqlField] attributes to 
generate field names. 
6) This is just a placeholder. It is much better than "Console.WriteLine" 
scattered all over the codebase that we had before. There are separate tasks 
for logging, I just did not want to add more WriteLine calls.
7) AsQueryable is an existing extension method in System.Linq.Queryable. Our 
method should stand out, because it implies a totally different mechanism 
underneath.
8) Yes, we do, it provides a way to eliminate LINQ translation overhead. The 
design is the same as in Entity Framework and will be familiar to the users.


> .Net: Create LINQ adapter for queries.
> --
>
> Key: IGNITE-1630
> URL: https://issues.apache.org/jira/browse/IGNITE-1630
> Project: Ignite
>  Issue Type: Task
>  Components: platforms
>Affects Versions: ignite-1.4
>Reporter: Vladimir Ozerov
>Assignee: Pavel Tupitsyn
>Priority: Critical
> Fix For: 1.6
>
>
> SQL queries are one of the most frequently used features in data grids. And 
> .Net comes with a very nice LINQ concept. 
> * LINQ provider should come in a separate assembly
> * Make sure that assembly is included in binary distribution



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (IGNITE-1630) .Net: Create LINQ adapter for queries.

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173607#comment-15173607
 ] 

Pavel Tupitsyn edited comment on IGNITE-1630 at 3/1/16 11:26 AM:
-

1) Done
2) Done (except fields queries, where this is not possible)
3) Allows to specify Java type name to be used in SQL index, if the user wants 
to override default mapping
4) Improves performance of the fields query. Instead allocating a List 
and then unboxing the values from it, we can read fields directly as specific 
types.
LINQ provider uses this method when doing fields queries. I'd like to avoid 
adding another InternalsVisibleTo for LINQ project, so this has to be on public 
API. 

And by itself, this method is not only faster than old one, but also more 
usable.
{code}cache.QueryFields((r,c)=>new MyClass(r.ReadInt(), r.ReadString())){code} 
is much better than working with untyped list.

5) Ignite SQL has reserved "_key" and "_val" field names to query cache entry 
key or value as a whole. LINQ provider looks at [QuerySqlField] attributes to 
generate field names. 
6) This is just a placeholder. It is much better than "Console.WriteLine" 
scattered all over the codebase that we had before. There are separate tasks 
for logging, I just did not want to add more WriteLine calls.
7) AsQueryable is an existing extension method in System.Linq.Queryable. Our 
method should stand out, because it implies a totally different mechanism 
underneath.
8) Yes, we do, it provides a way to eliminate LINQ translation overhead. The 
design is the same as in Entity Framework and will be familiar to the users.



was (Author: ptupitsyn):
1) Done
2) Done (except fields queries, where this is not possible)
3) Allows to specify Java type name to be used in SQL index, if the user wants 
to override default mapping
4) Improves performance of the fields query. Instead allocating a List 
and then unboxing the values from it, we can read fields directly as specific 
types.
LINQ provider uses this method when doing fields queries. I'd like to avoid 
adding another InternalsVisibleTo for LINQ project, so this has to be on public 
API. 

And by itself, this method is not only faster than old one, but also more 
usable.
{code}cache.QueryFields((r,c)=>new MyClass(r.ReadInt(), r.ReadString())){code} 
is much better than working with untyped list.

5) Ignite SQL has reserved "_key" and "_val" field names to query cache entry 
key or value as a whole. LINQ provider looks at [QuerySqlField] attributes to 
generate field names. 

> .Net: Create LINQ adapter for queries.
> --
>
> Key: IGNITE-1630
> URL: https://issues.apache.org/jira/browse/IGNITE-1630
> Project: Ignite
>  Issue Type: Task
>  Components: platforms
>Affects Versions: ignite-1.4
>Reporter: Vladimir Ozerov
>Assignee: Pavel Tupitsyn
>Priority: Critical
> Fix For: 1.6
>
>
> SQL queries are one of the most frequently used features in data grids. And 
> .Net comes with a very nice LINQ concept. 
> * LINQ provider should come in a separate assembly
> * Make sure that assembly is included in binary distribution



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (IGNITE-1630) .Net: Create LINQ adapter for queries.

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173607#comment-15173607
 ] 

Pavel Tupitsyn edited comment on IGNITE-1630 at 3/1/16 11:20 AM:
-

1) Done
2) Done (except fields queries, where this is not possible)
3) Allows to specify Java type name to be used in SQL index, if the user wants 
to override default mapping
4) Improves performance of the fields query. Instead allocating a List 
and then unboxing the values from it, we can read fields directly as specific 
types.
LINQ provider uses this method when doing fields queries. I'd like to avoid 
adding another InternalsVisibleTo for LINQ project, so this has to be on public 
API. 

And by itself, this method is not only faster than old one, but also more 
usable.
{code}cache.QueryFields((r,c)=>new MyClass(r.ReadInt(), r.ReadString())){code} 
is much better than working with untyped list.

5) Ignite SQL has reserved "_key" and "_val" field names to query cache entry 
key or value as a whole. LINQ provider looks at [QuerySqlField] attributes to 
generate field names. 


was (Author: ptupitsyn):
1) Done
2) Done (except fields queries, where this is not possible)
3) Allows to specify Java type name to be used in SQL index, if the user wants 
to override default mapping
4) Improves performance of the fields query. Instead allocating a List 
and then unboxing the values from it, we can read fields directly as specific 
types.
LINQ provider uses this method when doing fields queries. I'd like to avoid 
adding another InternalsVisibleTo for LINQ project, so this has to be on public 
API. 

And by itself, this method is not only faster than old one, but also more 
usable.
{code}cache.QueryFields((r,c)=>new MyClass(r.ReadInt(), r.ReadString())){code} 
is much better than working with untyped list.


> .Net: Create LINQ adapter for queries.
> --
>
> Key: IGNITE-1630
> URL: https://issues.apache.org/jira/browse/IGNITE-1630
> Project: Ignite
>  Issue Type: Task
>  Components: platforms
>Affects Versions: ignite-1.4
>Reporter: Vladimir Ozerov
>Assignee: Pavel Tupitsyn
>Priority: Critical
> Fix For: 1.6
>
>
> SQL queries are one of the most frequently used features in data grids. And 
> .Net comes with a very nice LINQ concept. 
> * LINQ provider should come in a separate assembly
> * Make sure that assembly is included in binary distribution



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (IGNITE-1630) .Net: Create LINQ adapter for queries.

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173607#comment-15173607
 ] 

Pavel Tupitsyn edited comment on IGNITE-1630 at 3/1/16 11:14 AM:
-

1) Done
2) Done (except fields queries, where this is not possible)
3) Allows to specify Java type name to be used in SQL index, if the user wants 
to override default mapping
4) Improves performance of the fields query. Instead allocating a List 
and then unboxing the values from it, we can read fields directly as specific 
types.
LINQ provider uses this method when doing fields queries. I'd like to avoid 
adding another InternalsVisibleTo for LINQ project, so this has to be on public 
API. 

And by itself, this method is not only faster than old one, but also more 
usable.
{code}cache.QueryFields((r,c)=>new MyClass(r.ReadInt(), r.ReadString())){code} 
is much better than working with untyped list.



was (Author: ptupitsyn):
1) Done
2) Done (except fields queries, where this is not possible)
3) Allows to specify Java type name to be used in SQL index, if the user wants 
to override default mapping
4) Improves performance of the fields query. Instead allocating a List 
and then unboxing the values from it, we can read fields directly as specific 
types.
LINQ provider uses this method when doing fields queries. I'd like to avoid 
adding another InternalsVisibleTo for LINQ project, so this has to be on public 
API. 


> .Net: Create LINQ adapter for queries.
> --
>
> Key: IGNITE-1630
> URL: https://issues.apache.org/jira/browse/IGNITE-1630
> Project: Ignite
>  Issue Type: Task
>  Components: platforms
>Affects Versions: ignite-1.4
>Reporter: Vladimir Ozerov
>Assignee: Pavel Tupitsyn
>Priority: Critical
> Fix For: 1.6
>
>
> SQL queries are one of the most frequently used features in data grids. And 
> .Net comes with a very nice LINQ concept. 
> * LINQ provider should come in a separate assembly
> * Make sure that assembly is included in binary distribution



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-2719) Value is not copied in entry processor if optimized marshaller is used

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173610#comment-15173610
 ] 

ASF GitHub Bot commented on IGNITE-2719:


Github user agura closed the pull request at:

https://github.com/apache/ignite/pull/520


> Value is not copied in entry processor if optimized marshaller is used
> --
>
> Key: IGNITE-2719
> URL: https://issues.apache.org/jira/browse/IGNITE-2719
> Project: Ignite
>  Issue Type: Bug
>  Components: cache
>Reporter: Valentin Kulichenko
>Assignee: Andrey Gura
>Priority: Blocker
>  Labels: community, important
> Fix For: 1.6
>
> Attachments: CacheEntryProcessorCopySelfTest.java
>
>
> If {{OptimizedMarshaller}} is used, the {{MutableEntry}} passed to entry 
> processor contains the same instance that is stored in cache, even if 
> {{copyOnRead}} flag is true.
> This happens because {{CacheLazyEntry.getValue()}} method never creates a 
> copy.
> Test attached.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-1630) .Net: Create LINQ adapter for queries.

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173607#comment-15173607
 ] 

Pavel Tupitsyn commented on IGNITE-1630:


1) Done
2) Done (except fields queries, where this is not possible)
3) Allows to specify Java type name to be used in SQL index, if the user wants 
to override default mapping
4) Improves performance of the fields query. Instead allocating a List 
and then unboxing the values from it, we can read fields directly as specific 
types.
LINQ provider uses this method when doing fields queries. I'd like to avoid 
adding another InternalsVisibleTo for LINQ project, so this has to be on public 
API. 


> .Net: Create LINQ adapter for queries.
> --
>
> Key: IGNITE-1630
> URL: https://issues.apache.org/jira/browse/IGNITE-1630
> Project: Ignite
>  Issue Type: Task
>  Components: platforms
>Affects Versions: ignite-1.4
>Reporter: Vladimir Ozerov
>Assignee: Pavel Tupitsyn
>Priority: Critical
> Fix For: 1.6
>
>
> SQL queries are one of the most frequently used features in data grids. And 
> .Net comes with a very nice LINQ concept. 
> * LINQ provider should come in a separate assembly
> * Make sure that assembly is included in binary distribution



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-2719) Value is not copied in entry processor if optimized marshaller is used

[Andrey Gura (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173587#comment-15173587
 ] 

Andrey Gura commented on IGNITE-2719:
-

Nodes start/stop moved to {{doTest}} method.

> Value is not copied in entry processor if optimized marshaller is used
> --
>
> Key: IGNITE-2719
> URL: https://issues.apache.org/jira/browse/IGNITE-2719
> Project: Ignite
>  Issue Type: Bug
>  Components: cache
>Reporter: Valentin Kulichenko
>Assignee: Andrey Gura
>Priority: Blocker
>  Labels: community, important
> Fix For: 1.6
>
> Attachments: CacheEntryProcessorCopySelfTest.java
>
>
> If {{OptimizedMarshaller}} is used, the {{MutableEntry}} passed to entry 
> processor contains the same instance that is stored in cache, even if 
> {{copyOnRead}} flag is true.
> This happens because {{CacheLazyEntry.getValue()}} method never creates a 
> copy.
> Test attached.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-1903) CacheStore implementation is serialised to grid clients whether they require it or not

[Michael Griggs (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-1903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173578#comment-15173578
 ] 

Michael Griggs commented on IGNITE-1903:


Denis, I'd like to look in to how we can solve this problem.  Do you have a 
preferred approach?

> CacheStore implementation is serialised to grid clients whether they require 
> it or not
> --
>
> Key: IGNITE-1903
> URL: https://issues.apache.org/jira/browse/IGNITE-1903
> Project: Ignite
>  Issue Type: Bug
>Affects Versions: 1.5.0.final
>Reporter: Michael Griggs
>Assignee: Denis Magda
>Priority: Critical
> Fix For: 1.6
>
>
> See User discussion thread:  
> http://apache-ignite-users.70518.x6.nabble.com/CacheStore-being-serialized-to-client-td1931.html
> Brief summary:  When a grid client joins the grid (clientMode=true) it 
> receives a message from the server node(s) on the grid that contains the 
> serialized CacheStore implementation object.  If the client does not have 
> this class on its CLASSPATH (and there is no reason it should, as it is a 
> client) then the de-serialization of this message will fail, causing this 
> exception:
> {code}SEVERE: Failed to unmarshal discovery data for component: 1 
> class org.apache.ignite.IgniteCheckedException: Failed to find class with 
> given class loader for unmarshalling (make sure same versions of all classes 
> are available on all nodes or enable peer-class-loading): 
> sun.misc.Launcher$AppClassLoader@14dad5dc 
> at 
> org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal(JdkMarshaller.java:104)
>  
> at 
> org.apache.ignite.marshaller.AbstractMarshaller.unmarshal(AbstractMarshaller.java:67)
>  
> at 
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.onExchange(TcpDiscoverySpi.java:1529)
>  
> at 
> org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.processNodeAddFinishedMessage(ClientImpl.java:1317)
>  
> at 
> org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.processDiscoveryMessage(ClientImpl.java:1229)
>  
> at 
> org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1199)
>  
> at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) 
> Caused by: java.lang.ClassNotFoundException: 
> c.g.r.cachewrapper.ignite.CacheMissHandlerIgnite
> {code}
> where {{c.g.r.cachewrapper.ignite.CacheMissHandlerIgnite}} is the CacheStore 
> implementation.
> The ostensible reason for the CacheStore serialization is so that clients of 
> a TRANSACTIONAL cache can begin the transaction on the underlying store.  
> The only current solution to this is to add the grid node's CacheStore 
> implementation class definition to the CLASSPATH of the client.  This creates 
> an *undesirable coupling* between server and client.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-1419) .Net: Add optional "raw" flag to binary type configuration.

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173574#comment-15173574
 ] 

Pavel Tupitsyn commented on IGNITE-1419:


1) This property should be readonly, that's why ctor arg is necessary. If the 
property is writeable, and user sets it after type registration, it won't take 
effect.
The "too many ctor args" problem has other solutions (introduce a struct to 
hold args).
2) Done

> .Net: Add optional "raw" flag to binary type configuration.
> ---
>
> Key: IGNITE-1419
> URL: https://issues.apache.org/jira/browse/IGNITE-1419
> Project: Ignite
>  Issue Type: Task
>  Components: platforms
>Affects Versions: 1.1.4
>Reporter: Vladimir Ozerov
>Assignee: Pavel Tupitsyn
> Fix For: 1.6
>
>
> By default we write .Net objects with field names included. But in some cases 
> it is not needed. E.g. when objects are not going to be used in queries.
> Removing field names will speed up serialization/deserialization. We can add 
> optional "raw" flag to portable type descriptor to control this behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (IGNITE-2702) .NET: Support compact footers in binary format.

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173544#comment-15173544
 ] 

Pavel Tupitsyn edited comment on IGNITE-2702 at 3/1/16 10:19 AM:
-

Performance check:
* BinarizableWriteBenchmark: 10% faster
* BinarizableReadBenchmark: unchanged


was (Author: ptupitsyn):
Performance check:
* BinarizableWriteBenchmark: 10% faster
* BinarizableReadBenchmark: no changes

> .NET: Support compact footers in binary format.
> ---
>
> Key: IGNITE-2702
> URL: https://issues.apache.org/jira/browse/IGNITE-2702
> Project: Ignite
>  Issue Type: Task
>  Components: platforms
>Affects Versions: 1.5.0.final
>Reporter: Vladimir Ozerov
>Assignee: Pavel Tupitsyn
>Priority: Critical
> Fix For: 1.6
>
>
> There is an optimization in binary format called "compact footers". When 
> enabled, we do not write field IDs during serialization and only write field 
> offsets. And field IDs are written only once to type metadata.
> We need to support the same mechanism in .NET.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-2702) .NET: Support compact footers in binary format.

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173544#comment-15173544
 ] 

Pavel Tupitsyn commented on IGNITE-2702:


Performance check:
* BinarizableWriteBenchmark: 10% faster
* BinarizableReadBenchmark: no changes

> .NET: Support compact footers in binary format.
> ---
>
> Key: IGNITE-2702
> URL: https://issues.apache.org/jira/browse/IGNITE-2702
> Project: Ignite
>  Issue Type: Task
>  Components: platforms
>Affects Versions: 1.5.0.final
>Reporter: Vladimir Ozerov
>Assignee: Pavel Tupitsyn
>Priority: Critical
> Fix For: 1.6
>
>
> There is an optimization in binary format called "compact footers". When 
> enabled, we do not write field IDs during serialization and only write field 
> offsets. And field IDs are written only once to type metadata.
> We need to support the same mechanism in .NET.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (IGNITE-2724) Implement configuration of ZooKeeper Ip finder

[Vasiliy Sisko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vasiliy Sisko resolved IGNITE-2724.
---
Resolution: Fixed
  Assignee: Pavel Konstantinov  (was: Vasiliy Sisko)

1-3 Fixed

> Implement configuration of ZooKeeper Ip finder
> --
>
> Key: IGNITE-2724
> URL: https://issues.apache.org/jira/browse/IGNITE-2724
> Project: Ignite
>  Issue Type: Sub-task
>  Components: wizards
>Affects Versions: 1.6
>Reporter: Vasiliy Sisko
>Assignee: Pavel Konstantinov
> Fix For: 1.6
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (IGNITE-2650) Ignite should throw an exception on start of dynamic cache with swap if Ignite uses NoopSwapSpaceSpi

[Anton Vinogradov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Vinogradov closed IGNITE-2650.


> Ignite should throw an exception on start of dynamic cache with swap if 
> Ignite uses NoopSwapSpaceSpi
> 
>
> Key: IGNITE-2650
> URL: https://issues.apache.org/jira/browse/IGNITE-2650
> Project: Ignite
>  Issue Type: Bug
>Affects Versions: 1.6
>Reporter: Artem Shutak
>Assignee: Konstantin Margorin
>  Labels: newbie
> Fix For: 1.6
>
>
> Ignite should throw an exception on start of dynamic cache with enabled swap 
> if Ignite uses NoopSwapSpaceSpi.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-2650) Ignite should throw an exception on start of dynamic cache with swap if Ignite uses NoopSwapSpaceSpi

[Anton Vinogradov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173475#comment-15173475
 ] 

Anton Vinogradov commented on IGNITE-2650:
--

Merged,
Thanks for contribution.

> Ignite should throw an exception on start of dynamic cache with swap if 
> Ignite uses NoopSwapSpaceSpi
> 
>
> Key: IGNITE-2650
> URL: https://issues.apache.org/jira/browse/IGNITE-2650
> Project: Ignite
>  Issue Type: Bug
>Affects Versions: 1.6
>Reporter: Artem Shutak
>Assignee: Konstantin Margorin
>  Labels: newbie
> Fix For: 1.6
>
>
> Ignite should throw an exception on start of dynamic cache with enabled swap 
> if Ignite uses NoopSwapSpaceSpi.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-2650) Ignite should throw an exception on start of dynamic cache with swap if Ignite uses NoopSwapSpaceSpi

[Anton Vinogradov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173461#comment-15173461
 ] 

Anton Vinogradov commented on IGNITE-2650:
--

Konstantin, 

Looks good to me, I've fixed minor codestyle issues and added couple of tests.
Please review my changes and I'll merge them to master.
https://github.com/apache/ignite/pull/526

> Ignite should throw an exception on start of dynamic cache with swap if 
> Ignite uses NoopSwapSpaceSpi
> 
>
> Key: IGNITE-2650
> URL: https://issues.apache.org/jira/browse/IGNITE-2650
> Project: Ignite
>  Issue Type: Bug
>Affects Versions: 1.6
>Reporter: Artem Shutak
>Assignee: Konstantin Margorin
>  Labels: newbie
> Fix For: 1.6
>
>
> Ignite should throw an exception on start of dynamic cache with enabled swap 
> if Ignite uses NoopSwapSpaceSpi.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-2738) Ignite-YARN custom queue name

[Luca Rea (JIRA)]

Luca Rea created IGNITE-2738:


 Summary: Ignite-YARN custom queue name
 Key: IGNITE-2738
 URL: https://issues.apache.org/jira/browse/IGNITE-2738
 Project: Ignite
  Issue Type: Bug
Reporter: Luca Rea


Hi,
Actually is not possible to pass a custom queue name when launching Ignite 
YARN, so all Ignite YARN containers always run into the "default" queue (that 
usually is configured to have limited resources available).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-2737) Ignite-Spark documentation is missing some useful informations

[Luca Rea (JIRA)]

Luca Rea created IGNITE-2737:


 Summary: Ignite-Spark documentation is missing some useful 
informations
 Key: IGNITE-2737
 URL: https://issues.apache.org/jira/browse/IGNITE-2737
 Project: Ignite
  Issue Type: Bug
Reporter: Luca Rea


Hi,

in my tests I have experienced some configuration issue running spark in local 
and yarn-client mode, so I want to share them.
In order to let Ignite work correctly I had to customize spark-defaults.conf 
adding to "spark.driver.extraClassPath" and "spark.executor.extraClassPath" the 
string
{code}
"/opt/ignite/libs/*:/opt/ignite/libs/optional/ignite-spark/*:/opt/ignite/libs/optional/ignite-log4j/*:/opt/ignite/libs/optional/ignite-yarn/*:/opt/ignite/libs/ignite-spring/*"
{code}
(opt/ignite is my IGNITE_HOME) and other IGNITE_ useful variables like 
"spark.executorEnv.IGNITE_WORK_DIR" in order to let them be loaded by executors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-2736) custom Ignite Configuration (not xml file) is not used by spark executors

[Luca Rea (JIRA)]

Luca Rea created IGNITE-2736:


 Summary: custom Ignite Configuration (not xml file) is not used by 
spark executors
 Key: IGNITE-2736
 URL: https://issues.apache.org/jira/browse/IGNITE-2736
 Project: Ignite
  Issue Type: Bug
Reporter: Luca Rea


Hi,
I have launched an Ignite Cluster inside YARN and I use spark-shell from a 
client to attach to existing cluster cluster in client mode, connection between 
client and cluster doesn't support multicast so I've tried to use a custom 
config like below:

{code}
import org.apache.ignite.spark._
import org.apache.ignite.configuration._
import org.apache.ignite._
import org.apache.ignite.spi.discovery.tcp._
val spi = new TcpDiscoverySpi();
import org.apache.ignite.spi.discovery.tcp.ipfinder.vm._
val ipFinder = new TcpDiscoveryVmIpFinder();
import java.util.Arrays
ipFinder.setAddresses(Arrays.asList("172.16.24.48:47500", "172.16.24.49:47500", 
"172.16.24.50:47500", "172.16.24.51:47500", "172.16.24.52:47500", 
"172.16.24.53:47500"));
spi.setIpFinder(ipFinder);
val cfg = new IgniteConfiguration() with Serializable;
cfg.setGridName("ignite-cluster");
cfg.setDiscoverySpi(spi);
val cacheCfg = new CacheConfiguration("myCache");
import org.apache.ignite.cache._
cacheCfg.setCacheMode(CacheMode.PARTITIONED);
cacheCfg.setBackups(1);
cfg.setCacheConfiguration(cacheCfg);
class cfg extends Serializable;

val ic = new IgniteContext[Integer, Integer](sc, () => cfg )

val sharedRdd = ic.fromCache("example")
val x = sqlContext.sparkContext.parallelize(1 to 1, 10).map(i => (new 
Integer(i), new Integer(i)))
sharedRdd.savePairs(x)
{code}

when I run the last command it freeze waiting to connect to the cluster, in 
fact it seems that in this way the spark execurors don't use the above 
configuration nor load the file default-config.xml but use some hardocoded 
configuration with only multicast enabled.

The workaround is to use a acustom xml configuration file and copy it into the 
config ignite path of all all spark nodes the run:

{code}
import org.apache.ignite.spark._
val ic = new IgniteContext[Integer, Integer](sc, "config/custom-config.xml")
{code}

custom-config.xml:
{code}
http://www.springframework.org/schema/beans;
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
   xsi:schemaLocation="
   http://www.springframework.org/schema/beans
   http://www.springframework.org/schema/beans/spring-beans.xsd;>



  
  

  

  

  172.16.24.48:47500
  172.16.24.49:47500
  172.16.24.50:47500
  172.16.24.51:47500
  172.16.24.52:47500
  172.16.24.53:47500

  

  

  



{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)