[jira] [Created] (IGNITE-2706) GridHandleTable stores object at incorrect index
Denis Magda created IGNITE-2706: --- Summary: GridHandleTable stores object at incorrect index Key: IGNITE-2706 URL: https://issues.apache.org/jira/browse/IGNITE-2706 Project: Ignite Issue Type: Bug Affects Versions: 1.5.0.final Reporter: Denis Magda Assignee: Valentin Kulichenko Priority: Blocker Fix For: 1.6 Attachments: GridHandleTableUnitTest.java When lookup(Object o) calls growSpine() it uses OLD index to insert object. Test with additional details is attached. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: OSGi Support?
Looks like the sun.nio.ch package is included in OpenJDK: http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/tip/src/share/classes/sun/nio/ch/ServerSocketChannelImpl.java. But my bet is that Karaf doesn't export it by default because it is not part of the standard JRE spec. Raúl. On 20 Feb 2016 20:01, "Dmitriy Setrakyan" wrote: > Raul, > > To my knowledge Ignite can run under OpenJDK as well. Would this package be > required in OpenJDK? > > D. > > On Sat, Feb 20, 2016 at 11:24 AM, Raul Kripalani wrote: > > > Hey John, > > > > Yeah, this is expected. Ignite uses some packages only available in the > > Oracle JDK (proprietary Sun packages). > > > > Karaf doesn't export the sun.nio.ch package by default from bundle 0 > > (system bundle). > > > > You need to add it to the jre.properties file manually in the Karaf etc > > directory. Then you restart Karaf and it should be exported by the system > > bundle (0). > > > > I'll add these instructions to the Ignite Docs, and also to the Camel > wiki, > > over the weekend. > > > > Thanks for digging into this, John. I had missed the Camel ticket. > > > > Cheers, > > Raúl. > > On 20 Feb 2016 15:03, "John D. Ament" wrote: > > > > > Hey guys > > > > > > I was looking at a Camel issue this morning. > > > > > > https://issues.apache.org/jira/browse/CAMEL-9458 > > > > > > It appears that camel-ignite doesn't install properly in Karaf. > Looking > > at > > > it further, it seems that Ignite's OSGi module doesn't install in > Karaf, > > > directly. I've copied the output from that ticket here for reference > > > > > > *karaf*@root()> feature:repo-add ignite 1.5.0.final > > > > > > Adding feature url > > > mvn:org.apache.ignite/ignite-osgi-karaf/1.5.0.final/xml/features > > > > > > *karaf*@root()> feature:install ignite-core > > > > > > Error executing command: Unable to resolve root: missing requirement > > [root] > > > osgi.identity; osgi.identity=ignite-core; type=karaf.feature; > > > version="[1.5.0.final,1.5.0.final]"; > > > > > > > > > filter:="(&(osgi.identity=ignite-core)(type=karaf.feature)(version>=1.5.0.final)(version<=1.5.0.final))" > > > [caused by: Unable to resolve ignite-core/1.5.0.final: missing > > requirement > > > [ignite-core/1.5.0.final] osgi.identity; > > > osgi.identity=org.apache.ignite.ignite-core; type=osgi.bundle; > > > version="[1.5.0.final,1.5.0.final]"; resolution:=mandatory [caused by: > > > Unable to resolve org.apache.ignite.ignite-core/1.5.0.final: missing > > > requirement [org.apache.ignite.ignite-core/1.5.0.final] > > > osgi.wiring.package; filter:="(osgi.wiring.package=sun.nio.ch)"]] > > > > > > > > > So I was wondering, is your OSGi manifest valid? > > > > > > > > > John > > > > > >
[jira] [Created] (IGNITE-2705) .NET: Add a test to verify ToolsVersion in project and solution files
Pavel Tupitsyn created IGNITE-2705: -- Summary: .NET: Add a test to verify ToolsVersion in project and solution files Key: IGNITE-2705 URL: https://issues.apache.org/jira/browse/IGNITE-2705 Project: Ignite Issue Type: Test Components: platforms Affects Versions: 1.1.4 Reporter: Pavel Tupitsyn Assignee: Pavel Tupitsyn Fix For: 1.6 We maintain VS2010 support, engineers work in newer versions, which often causes incorrect ToolsVersion in csproj and sln files. Add a test to scan all files and verify tools version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Apache Arrow and Apache Ignite
Agree. Feel free to initiate a discussion on the Arrow list and cross-copy this list. I will also think about it some more and will propose something next week. On Sat, Feb 20, 2016 at 12:13 PM, Dood@ODDO wrote: > That's the million dollar question - I think we should approach the Arrow > people and get a conversation going. We want to be ahead of the curve, not > behind it - Arrow seems to be making quite a stir, not to mention that it > was fast-tracked to mature project status apparently solely based on the > people/companies involved ;-) > > > On 2/20/2016 11:14 AM, Dmitriy Setrakyan wrote: > >> It sounds to me that it does not store it’s own data, but allows to >> process >> data from somewhere else, right? If yes, will I be able to take Ignite >> data >> with Arrow and convert it to a columnar format for some other processing? >> >> On Sat, Feb 20, 2016 at 2:48 AM, Roman Shtykh >> wrote: >> >> It's a standard (including format and algorithms) to share in-memory data >>> between several big data platforms. >>> In my understanding, it is like placing data in in-memory columns in >>> Parquet and being able to use it in Kudu without >>> serializing/deserializing. >>> >>> -Roman >>> >>> >
Re: Apache Arrow and Apache Ignite
That's the million dollar question - I think we should approach the Arrow people and get a conversation going. We want to be ahead of the curve, not behind it - Arrow seems to be making quite a stir, not to mention that it was fast-tracked to mature project status apparently solely based on the people/companies involved ;-) On 2/20/2016 11:14 AM, Dmitriy Setrakyan wrote: It sounds to me that it does not store it’s own data, but allows to process data from somewhere else, right? If yes, will I be able to take Ignite data with Arrow and convert it to a columnar format for some other processing? On Sat, Feb 20, 2016 at 2:48 AM, Roman Shtykh wrote: It's a standard (including format and algorithms) to share in-memory data between several big data platforms. In my understanding, it is like placing data in in-memory columns in Parquet and being able to use it in Kudu without serializing/deserializing. -Roman
Re: OSGi Support?
Raul, To my knowledge Ignite can run under OpenJDK as well. Would this package be required in OpenJDK? D. On Sat, Feb 20, 2016 at 11:24 AM, Raul Kripalani wrote: > Hey John, > > Yeah, this is expected. Ignite uses some packages only available in the > Oracle JDK (proprietary Sun packages). > > Karaf doesn't export the sun.nio.ch package by default from bundle 0 > (system bundle). > > You need to add it to the jre.properties file manually in the Karaf etc > directory. Then you restart Karaf and it should be exported by the system > bundle (0). > > I'll add these instructions to the Ignite Docs, and also to the Camel wiki, > over the weekend. > > Thanks for digging into this, John. I had missed the Camel ticket. > > Cheers, > Raúl. > On 20 Feb 2016 15:03, "John D. Ament" wrote: > > > Hey guys > > > > I was looking at a Camel issue this morning. > > > > https://issues.apache.org/jira/browse/CAMEL-9458 > > > > It appears that camel-ignite doesn't install properly in Karaf. Looking > at > > it further, it seems that Ignite's OSGi module doesn't install in Karaf, > > directly. I've copied the output from that ticket here for reference > > > > *karaf*@root()> feature:repo-add ignite 1.5.0.final > > > > Adding feature url > > mvn:org.apache.ignite/ignite-osgi-karaf/1.5.0.final/xml/features > > > > *karaf*@root()> feature:install ignite-core > > > > Error executing command: Unable to resolve root: missing requirement > [root] > > osgi.identity; osgi.identity=ignite-core; type=karaf.feature; > > version="[1.5.0.final,1.5.0.final]"; > > > > > filter:="(&(osgi.identity=ignite-core)(type=karaf.feature)(version>=1.5.0.final)(version<=1.5.0.final))" > > [caused by: Unable to resolve ignite-core/1.5.0.final: missing > requirement > > [ignite-core/1.5.0.final] osgi.identity; > > osgi.identity=org.apache.ignite.ignite-core; type=osgi.bundle; > > version="[1.5.0.final,1.5.0.final]"; resolution:=mandatory [caused by: > > Unable to resolve org.apache.ignite.ignite-core/1.5.0.final: missing > > requirement [org.apache.ignite.ignite-core/1.5.0.final] > > osgi.wiring.package; filter:="(osgi.wiring.package=sun.nio.ch)"]] > > > > > > So I was wondering, is your OSGi manifest valid? > > > > > > John > > >
Re: OSGi Support?
Hey John, Yeah, this is expected. Ignite uses some packages only available in the Oracle JDK (proprietary Sun packages). Karaf doesn't export the sun.nio.ch package by default from bundle 0 (system bundle). You need to add it to the jre.properties file manually in the Karaf etc directory. Then you restart Karaf and it should be exported by the system bundle (0). I'll add these instructions to the Ignite Docs, and also to the Camel wiki, over the weekend. Thanks for digging into this, John. I had missed the Camel ticket. Cheers, Raúl. On 20 Feb 2016 15:03, "John D. Ament" wrote: > Hey guys > > I was looking at a Camel issue this morning. > > https://issues.apache.org/jira/browse/CAMEL-9458 > > It appears that camel-ignite doesn't install properly in Karaf. Looking at > it further, it seems that Ignite's OSGi module doesn't install in Karaf, > directly. I've copied the output from that ticket here for reference > > *karaf*@root()> feature:repo-add ignite 1.5.0.final > > Adding feature url > mvn:org.apache.ignite/ignite-osgi-karaf/1.5.0.final/xml/features > > *karaf*@root()> feature:install ignite-core > > Error executing command: Unable to resolve root: missing requirement [root] > osgi.identity; osgi.identity=ignite-core; type=karaf.feature; > version="[1.5.0.final,1.5.0.final]"; > > filter:="(&(osgi.identity=ignite-core)(type=karaf.feature)(version>=1.5.0.final)(version<=1.5.0.final))" > [caused by: Unable to resolve ignite-core/1.5.0.final: missing requirement > [ignite-core/1.5.0.final] osgi.identity; > osgi.identity=org.apache.ignite.ignite-core; type=osgi.bundle; > version="[1.5.0.final,1.5.0.final]"; resolution:=mandatory [caused by: > Unable to resolve org.apache.ignite.ignite-core/1.5.0.final: missing > requirement [org.apache.ignite.ignite-core/1.5.0.final] > osgi.wiring.package; filter:="(osgi.wiring.package=sun.nio.ch)"]] > > > So I was wondering, is your OSGi manifest valid? > > > John >
Re: Binary object inside Externalizable
Andrey, you are absolutely right. I was suggesting a quick fix, which in my view, would fix most of the issues. In the long term, we should fix the binary serialization to work the way you are describing. D. On Sat, Feb 20, 2016 at 9:26 AM, Andrey Kornev wrote: > Val, > > I think Ignite Serialization should follow the approach similar to what > Java serialization does: it can transparently and consistently handle both > Serializable and Externalizable classes. It can do it because it relies on > a single class - ObjectOutputStream - to do serialization (and a single > class - ObjectInputStream - to do the opposite, but let's ignore it for the > moment). ObjectOutputStream delegates to proper user callbacks (writeObject > or writeExternal) at the right moments, but otherwise it fully controls the > on-the-wire representation and takes care of all the necessary stream > framing/marking so that ObjectInputStream can then correctly deserialize > the bytes. It's never a problem for Java to serialize an Externalizable > field from a Serializable object and vice versa. Users can mix and match > serialization APIs as they please. > > I think Ignite should just have a single marshaller, let's say the Binary, > which drives the whole serialization process. It doesn't delegate to > OptimizedMarshaller as it does today. Instead it detects the > Serializable/Externalizable classes and calls their serialization callbacks > - writeObject/writeExternal - passing in *itself* as > ObjectOutputStream/ObjectOutput correspondingly. This way the entire > serialization process never leaves the context of a single Binary > marshaller and allows Ignite to handle such complex cases as the one we're > discussing now. > > I hope it makes sense. > > Andrey > > > From: valentin.kuliche...@gmail.com > > Date: Fri, 19 Feb 2016 19:15:07 -0800 > > Subject: Binary object inside Externalizable > > To: dev@ignite.apache.org > > > > Folks, > > > > I recently faced an issue which seems to be pretty critical. As you know, > > when binary marshaller meets an Externalizable object, it switches to > > optimized marshaller. And if there is a regular object inside, it's still > > serialized with optimized, even if its type is declared in binary > > configuration with custom mapper, etc. This looks wrong. > > > > It's getting even worse in compute grid, because closure processor wraps > > user's closures in some internal classes, which are Externalizable, so > > these closures are always serialized with optimized marshaller. > > > > Does anyone have ideas on what is the proper fix for this? > > > > -Val > >
RE: Binary object inside Externalizable
Val, I think Ignite Serialization should follow the approach similar to what Java serialization does: it can transparently and consistently handle both Serializable and Externalizable classes. It can do it because it relies on a single class - ObjectOutputStream - to do serialization (and a single class - ObjectInputStream - to do the opposite, but let's ignore it for the moment). ObjectOutputStream delegates to proper user callbacks (writeObject or writeExternal) at the right moments, but otherwise it fully controls the on-the-wire representation and takes care of all the necessary stream framing/marking so that ObjectInputStream can then correctly deserialize the bytes. It's never a problem for Java to serialize an Externalizable field from a Serializable object and vice versa. Users can mix and match serialization APIs as they please. I think Ignite should just have a single marshaller, let's say the Binary, which drives the whole serialization process. It doesn't delegate to OptimizedMarshaller as it does today. Instead it detects the Serializable/Externalizable classes and calls their serialization callbacks - writeObject/writeExternal - passing in *itself* as ObjectOutputStream/ObjectOutput correspondingly. This way the entire serialization process never leaves the context of a single Binary marshaller and allows Ignite to handle such complex cases as the one we're discussing now. I hope it makes sense. Andrey > From: valentin.kuliche...@gmail.com > Date: Fri, 19 Feb 2016 19:15:07 -0800 > Subject: Binary object inside Externalizable > To: dev@ignite.apache.org > > Folks, > > I recently faced an issue which seems to be pretty critical. As you know, > when binary marshaller meets an Externalizable object, it switches to > optimized marshaller. And if there is a regular object inside, it's still > serialized with optimized, even if its type is declared in binary > configuration with custom mapper, etc. This looks wrong. > > It's getting even worse in compute grid, because closure processor wraps > user's closures in some internal classes, which are Externalizable, so > these closures are always serialized with optimized marshaller. > > Does anyone have ideas on what is the proper fix for this? > > -Val
Re: Apache Arrow and Apache Ignite
It sounds to me that it does not store it’s own data, but allows to process data from somewhere else, right? If yes, will I be able to take Ignite data with Arrow and convert it to a columnar format for some other processing? On Sat, Feb 20, 2016 at 2:48 AM, Roman Shtykh wrote: > It's a standard (including format and algorithms) to share in-memory data > between several big data platforms. > In my understanding, it is like placing data in in-memory columns in > Parquet and being able to use it in Kudu without serializing/deserializing. > > -Roman >
Re: Binary object inside Externalizable
This would be a difficult issue to solve because of circular references. Can we in the mean time just make sure that none of Ignite internal objects are externalizable, so we don’t purposely force this situation onto our users? D. On Fri, Feb 19, 2016 at 7:15 PM, Valentin Kulichenko < valentin.kuliche...@gmail.com> wrote: > Folks, > > I recently faced an issue which seems to be pretty critical. As you know, > when binary marshaller meets an Externalizable object, it switches to > optimized marshaller. And if there is a regular object inside, it's still > serialized with optimized, even if its type is declared in binary > configuration with custom mapper, etc. This looks wrong. > > It's getting even worse in compute grid, because closure processor wraps > user's closures in some internal classes, which are Externalizable, so > these closures are always serialized with optimized marshaller. > > Does anyone have ideas on what is the proper fix for this? > > -Val >
Re: ODBC processor default behaviour
Not sure if it works. DB connection string should have certain port afaik. --Yakov 2016-02-20 18:26 GMT+03:00 Sergey Kozlov : > What's about to use the port range like TcpDiscoveryVmIpFinder? > > On Sat, Feb 20, 2016 at 6:22 PM, Yakov Zhdanov > wrote: > > > How about outputting warning like and keep retrying in a background > thread? > > > > warning - "Failed to bind ODBC processor TCP server to port (retrying > every > > 2 sec) [port=ABC] > > > > --Yakov > > > > 2016-02-20 17:38 GMT+03:00 Igor Sapego : > > > > > Igniters, > > > > > > I'm currently working on the ODBC driver. It connects by TCP to the > > > OdbcProcessor > > > on the node side. OdbcProcessor is enabled by default and it starts TCP > > > server on the > > > specific TCP port or throws exception if the port is busy. > > > > > > The problem is that such behavior breaks tests that start more than one > > > node on the > > > same machine as the second node fails to start because ODBC port is > > already > > > taken > > > by the first node. > > > > > > Does anyone has ideas what is the best way to fix that? > > > > > > Best Regards, > > > Igor > > > > > > > > > -- > Sergey Kozlov > GridGain Systems > www.gridgain.com >
Re: ODBC processor default behaviour
What's about to use the port range like TcpDiscoveryVmIpFinder? On Sat, Feb 20, 2016 at 6:22 PM, Yakov Zhdanov wrote: > How about outputting warning like and keep retrying in a background thread? > > warning - "Failed to bind ODBC processor TCP server to port (retrying every > 2 sec) [port=ABC] > > --Yakov > > 2016-02-20 17:38 GMT+03:00 Igor Sapego : > > > Igniters, > > > > I'm currently working on the ODBC driver. It connects by TCP to the > > OdbcProcessor > > on the node side. OdbcProcessor is enabled by default and it starts TCP > > server on the > > specific TCP port or throws exception if the port is busy. > > > > The problem is that such behavior breaks tests that start more than one > > node on the > > same machine as the second node fails to start because ODBC port is > already > > taken > > by the first node. > > > > Does anyone has ideas what is the best way to fix that? > > > > Best Regards, > > Igor > > > -- Sergey Kozlov GridGain Systems www.gridgain.com
Re: ODBC processor default behaviour
How about outputting warning like and keep retrying in a background thread? warning - "Failed to bind ODBC processor TCP server to port (retrying every 2 sec) [port=ABC] --Yakov 2016-02-20 17:38 GMT+03:00 Igor Sapego : > Igniters, > > I'm currently working on the ODBC driver. It connects by TCP to the > OdbcProcessor > on the node side. OdbcProcessor is enabled by default and it starts TCP > server on the > specific TCP port or throws exception if the port is busy. > > The problem is that such behavior breaks tests that start more than one > node on the > same machine as the second node fails to start because ODBC port is already > taken > by the first node. > > Does anyone has ideas what is the best way to fix that? > > Best Regards, > Igor >
Re: OSGi Support?
Raul, can you please help with the issue? --Yakov 2016-02-20 18:03 GMT+03:00 John D. Ament : > Hey guys > > I was looking at a Camel issue this morning. > > https://issues.apache.org/jira/browse/CAMEL-9458 > > It appears that camel-ignite doesn't install properly in Karaf. Looking at > it further, it seems that Ignite's OSGi module doesn't install in Karaf, > directly. I've copied the output from that ticket here for reference > > *karaf*@root()> feature:repo-add ignite 1.5.0.final > > Adding feature url > mvn:org.apache.ignite/ignite-osgi-karaf/1.5.0.final/xml/features > > *karaf*@root()> feature:install ignite-core > > Error executing command: Unable to resolve root: missing requirement [root] > osgi.identity; osgi.identity=ignite-core; type=karaf.feature; > version="[1.5.0.final,1.5.0.final]"; > > filter:="(&(osgi.identity=ignite-core)(type=karaf.feature)(version>=1.5.0.final)(version<=1.5.0.final))" > [caused by: Unable to resolve ignite-core/1.5.0.final: missing requirement > [ignite-core/1.5.0.final] osgi.identity; > osgi.identity=org.apache.ignite.ignite-core; type=osgi.bundle; > version="[1.5.0.final,1.5.0.final]"; resolution:=mandatory [caused by: > Unable to resolve org.apache.ignite.ignite-core/1.5.0.final: missing > requirement [org.apache.ignite.ignite-core/1.5.0.final] > osgi.wiring.package; filter:="(osgi.wiring.package=sun.nio.ch)"]] > > > So I was wondering, is your OSGi manifest valid? > > > John >
OSGi Support?
Hey guys I was looking at a Camel issue this morning. https://issues.apache.org/jira/browse/CAMEL-9458 It appears that camel-ignite doesn't install properly in Karaf. Looking at it further, it seems that Ignite's OSGi module doesn't install in Karaf, directly. I've copied the output from that ticket here for reference *karaf*@root()> feature:repo-add ignite 1.5.0.final Adding feature url mvn:org.apache.ignite/ignite-osgi-karaf/1.5.0.final/xml/features *karaf*@root()> feature:install ignite-core Error executing command: Unable to resolve root: missing requirement [root] osgi.identity; osgi.identity=ignite-core; type=karaf.feature; version="[1.5.0.final,1.5.0.final]"; filter:="(&(osgi.identity=ignite-core)(type=karaf.feature)(version>=1.5.0.final)(version<=1.5.0.final))" [caused by: Unable to resolve ignite-core/1.5.0.final: missing requirement [ignite-core/1.5.0.final] osgi.identity; osgi.identity=org.apache.ignite.ignite-core; type=osgi.bundle; version="[1.5.0.final,1.5.0.final]"; resolution:=mandatory [caused by: Unable to resolve org.apache.ignite.ignite-core/1.5.0.final: missing requirement [org.apache.ignite.ignite-core/1.5.0.final] osgi.wiring.package; filter:="(osgi.wiring.package=sun.nio.ch)"]] So I was wondering, is your OSGi manifest valid? John
[GitHub] ignite pull request: IGNITE-2704: Disabled ODBC for core tests.
GitHub user isapego opened a pull request: https://github.com/apache/ignite/pull/507 IGNITE-2704: Disabled ODBC for core tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/isapego/ignite ignite-2704 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/507.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #507 commit 5ff555b12b74dc41a4a0fe1faaa2715c614d18a7 Author: isapego Date: 2016-02-20T14:58:33Z IGNITE-2704: Disabled ODBC for core tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (IGNITE-2704) CPP: Disable ODBC in tests configuration to prevent tests with multiple nodes from failing.
Igor Sapego created IGNITE-2704: --- Summary: CPP: Disable ODBC in tests configuration to prevent tests with multiple nodes from failing. Key: IGNITE-2704 URL: https://issues.apache.org/jira/browse/IGNITE-2704 Project: Ignite Issue Type: Task Components: clients Reporter: Igor Sapego Assignee: Igor Sapego {{OdbcConfiguration}} property of the {{IgniteConfiguration}} should be set to {{null}} for tests that are not require ODBC to pass. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] ignite pull request: IGNITE-2273
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/433 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
ODBC processor default behaviour
Igniters, I'm currently working on the ODBC driver. It connects by TCP to the OdbcProcessor on the node side. OdbcProcessor is enabled by default and it starts TCP server on the specific TCP port or throws exception if the port is busy. The problem is that such behavior breaks tests that start more than one node on the same machine as the second node fails to start because ODBC port is already taken by the first node. Does anyone has ideas what is the best way to fix that? Best Regards, Igor
[GitHub] ignite pull request: IGNITE-2700
GitHub user ilantukh opened a pull request: https://github.com/apache/ignite/pull/506 IGNITE-2700 You can merge this pull request into a Git repository by running: $ git pull https://github.com/ilantukh/ignite ignite-2700 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/506.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #506 commit 2fc88c57ebad5ea2ab5bd70f8fe7d34b900a0080 Author: Ilya Lantukh Date: 2016-02-20T11:46:53Z New versions of closures. commit 79fd06f1edb631b194d8c3b21e8c3adc9be88ea2 Author: Ilya Lantukh Date: 2016-02-20T13:04:47Z Fixed failing tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (IGNITE-2703) .NET: Dynamically registered classes must use binary serialization if possible.
Vladimir Ozerov created IGNITE-2703: --- Summary: .NET: Dynamically registered classes must use binary serialization if possible. Key: IGNITE-2703 URL: https://issues.apache.org/jira/browse/IGNITE-2703 Project: Ignite Issue Type: Task Components: platforms Affects Versions: 1.5.0.final Reporter: Vladimir Ozerov Assignee: Pavel Tupitsyn Priority: Critical Fix For: 1.6 At present we support dynamic class registration in .NET, but they are written using deafult .NET mechanism. This is counterintuitive for users and not consistent with Java, where such classes are written in binary form. Proposed implementation plan: 1) For each dynamically registered class we must understand whether it could be serialized through binary or not. If not - print a warning and fallback to .NET. 2) Before writing a class we must ensure that it's [typeId -> name] pair is known to the cluster. If not - write full class name instead of type ID. Java already do that. 3) Last, to support backward compatibility we must be able to fallback to current mode with help of some boolean flag. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (IGNITE-2702) .NET: Support compact footers in binary format.
Vladimir Ozerov created IGNITE-2702: --- Summary: .NET: Support compact footers in binary format. Key: IGNITE-2702 URL: https://issues.apache.org/jira/browse/IGNITE-2702 Project: Ignite Issue Type: Task Components: platforms Affects Versions: 1.5.0.final Reporter: Vladimir Ozerov Assignee: Pavel Tupitsyn Priority: Critical Fix For: 1.6 There is an optimization in binary format called "compact footers". When enabled, we do not write field IDs during serialization and only write field offsets. And field IDs are written only once to type metadata. We need to support the same mechanism in .NET. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Binary mode for Data Structures
No replies, so I've created an issue: https://issues.apache.org/jira/browse/IGNITE-2701 On Thu, Feb 18, 2016 at 3:06 PM, Pavel Tupitsyn wrote: > Igniters, > > Currently we don't have binary mode (withKeepBinary) in Data Structures > (Queue, AtomicReference). > Are there any plans for implementing this, or may be a workaround? > > Lack of binary mode prevents us from implementing IgniteQueue in .NET > (.NET types in most cases can't be deserialized in Java). > > AtomicReference is already implemented, and it works because values are > wrapped in GridCacheAtomicReferenceValue. So there is inconsistent behavior > between data structures. > > Thoughts? > > -- > -- > Pavel Tupitsyn > GridGain Systems, Inc. > www.gridgain.com > -- -- Pavel Tupitsyn GridGain Systems, Inc. www.gridgain.com
[jira] [Created] (IGNITE-2701) Binary mode in Data Structures
Pavel Tupitsyn created IGNITE-2701: -- Summary: Binary mode in Data Structures Key: IGNITE-2701 URL: https://issues.apache.org/jira/browse/IGNITE-2701 Project: Ignite Issue Type: Improvement Components: data structures Affects Versions: 1.1.4 Reporter: Pavel Tupitsyn Fix For: 1.6 Add binary mode (withKeepBinary) to Data Structures (Queue, AtomicReference). This will allow retrieving values in binary format when needed or when class is not available, and will allow implementing these structures in other platforms (.NET, C++). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (IGNITE-2700) GridClosureProcessor internal closures are [de]serialized by OptimizedMarshaller even if BinaryMarshaller is configured
Ilya Lantukh created IGNITE-2700: Summary: GridClosureProcessor internal closures are [de]serialized by OptimizedMarshaller even if BinaryMarshaller is configured Key: IGNITE-2700 URL: https://issues.apache.org/jira/browse/IGNITE-2700 Project: Ignite Issue Type: Bug Reporter: Ilya Lantukh Assignee: Ilya Lantukh -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] ignite pull request: IGNITE-2621 .NET: Fix compute in mixed-platfo...
GitHub user ptupitsyn opened a pull request: https://github.com/apache/ignite/pull/505 IGNITE-2621 .NET: Fix compute in mixed-platform cluster You can merge this pull request into a Git repository by running: $ git pull https://github.com/ptupitsyn/ignite ignite-2621 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/505.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #505 commit 4add7c20525b5a3e5a21e79332c22b158787bfc7 Author: Pavel Tupitsyn Date: 2016-02-19T11:12:39Z wip commit e24e9cbde4ebb9188af0d44764ac8a1601ff4743 Author: Pavel Tupitsyn Date: 2016-02-19T11:18:06Z Test done commit 3dc6551bbb6694a745bb23e0a543454f994f4156 Author: Pavel Tupitsyn Date: 2016-02-19T11:56:53Z wip commit 7299892aafb03117424aa78ab37719917267c889 Author: Pavel Tupitsyn Date: 2016-02-19T12:05:01Z wip commit 9c83264e9d7e8d2b007bf78c89bd8f2a654f3cb7 Author: Pavel Tupitsyn Date: 2016-02-19T12:15:28Z wip tests commit 9aabdaf73bbf97951753a59d9cd743f265b7ef4e Author: Pavel Tupitsyn Date: 2016-02-19T12:15:49Z wip commit 4a13673c2ff89a2f985d88e1fc671f2c93d8bd46 Author: Pavel Tupitsyn Date: 2016-02-19T13:07:31Z test done commit b9b43a356b387851dd3978b82d612b5b482af01e Author: Pavel Tupitsyn Date: 2016-02-19T13:26:37Z Fix process tree kill commit 51f380691a6898b46f47740f1f6a9e05b8baf590 Author: Pavel Tupitsyn Date: 2016-02-19T13:30:01Z Test done commit 09e9798c7c4adf4ab825ec5736eca258b95f677b Author: Pavel Tupitsyn Date: 2016-02-19T14:19:01Z Fixing platform compute projection commit 81d1cfde3dd421f603fc9141968666df36d095df Author: Pavel Tupitsyn Date: 2016-02-19T14:26:26Z Fixed platform compute projection commit b1533a5328a64334983f9d68c3c04da2c8312119 Author: Pavel Tupitsyn Date: 2016-02-19T14:32:02Z Merge remote-tracking branch 'remotes/upstream/master' into ignite-2621 commit 0dc28d31fbd29df6085c05157db8be83ea5acf94 Author: Pavel Tupitsyn Date: 2016-02-19T14:44:24Z wip tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: Apache Arrow and Apache Ignite
It's a standard (including format and algorithms) to share in-memory data between several big data platforms. In my understanding, it is like placing data in in-memory columns in Parquet and being able to use it in Kudu without serializing/deserializing. -Roman
[GitHub] ignite pull request: IGNITE-2579
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/500 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: Ignite instance hangs during restart in client mode
We can use some hacks (which do not seem to be a good solutions to me): 1. throw exception from startStopwatch() if start time has already been inited. Ugly since currently this method does not throw any error. 2. check if resources have already been injected to SPI before doing new injection. Not always working since SPI may not use injected resources. I think that each SPI should check for repeating start attempts itself. How about implementing spiStart() in SPI adapter and set some flag there or throw exception if it has already been set. All internal implementation will be changed to call super.spiStart() as first statement. Val, if you agree please file a newbie ticket. --Yakov 2016-02-20 1:55 GMT+03:00 Valentin Kulichenko : > Crossposting to dev@. > > Folks, > > This is not the first time I see this issue. Is there a way to detect it > and throw an exception? > > -Val > > On Fri, Feb 19, 2016 at 2:42 PM, vkulichenko < > valentin.kuliche...@gmail.com> wrote: > >> Hi Vinay, >> >> Yes, you should always create the configuration object from scratch when >> starting a node. Mainly it's because it contains SPIs (e.g., >> TcpDiscoverySpi) which have lifecycle and can't be reused. >> >> Makes sense? >> >> -Val >> >> >> >> -- >> View this message in context: >> http://apache-ignite-users.70518.x6.nabble.com/Ignite-instance-hangs-during-restart-in-client-mode-tp3101p3106.html >> Sent from the Apache Ignite Users mailing list archive at Nabble.com. >> > >
[jira] [Created] (IGNITE-2699) Starvation when CacheEntryListenerConfiguration.isSynchronous is true
Nikolay Tikhonov created IGNITE-2699: Summary: Starvation when CacheEntryListenerConfiguration.isSynchronous is true Key: IGNITE-2699 URL: https://issues.apache.org/jira/browse/IGNITE-2699 Project: Ignite Issue Type: Bug Components: cache Affects Versions: 1.5.0.final Reporter: Nikolay Tikhonov Fix For: 1.6 When event notifications in {{CacheEntryListener}} configured as synchronous possible situation that all thread in {{SYSTEM}} pool will be stuck on waiting ack messages. It is necessary to make the processing of acknowledge messages in another thread pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (IGNITE-2698) CacheKeyConfiguration is used only with BinaryMarshaller
Semen Boikov created IGNITE-2698: Summary: CacheKeyConfiguration is used only with BinaryMarshaller Key: IGNITE-2698 URL: https://issues.apache.org/jira/browse/IGNITE-2698 Project: Ignite Issue Type: Bug Components: cache Reporter: Semen Boikov Assignee: Yakov Zhdanov Fix For: 1.6 Now if CacheKeyConfiguration is specified it is used only with BinaryMarshaller. Need to take it into account with others marshallers as well or update documentation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] ignite pull request: Ignite 2598
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/480 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: NoopSwapSpaceSpi and dynamically started caches
Hello. Patch for https://issues.apache.org/jira/browse/IGNITE-2650 is available. Please review. On Thu, Feb 18, 2016 at 2:10 PM, Konstantin Margorin wrote: > Yakov, thank you. Sounds good. I'll do this way. > > On Thu, Feb 18, 2016 at 1:43 PM, Yakov Zhdanov > wrote: > >> Konstantin, >> >> Here are my comments. >> >> 1. I would force same swap configuration on all servers. Therefore, I >> would >> add "checkClient" attribute to annotation which is "true" by default >> 2. I think all users should properly configure swap along the cluster if >> they want to handle possible memory overflow. >> >> --Yakov >> >> 2016-02-18 12:35 GMT+03:00 Konstantin Margorin : >> >> > Hi all. >> > >> > Sorry for asking again, but after some experiments I want to discuss >> > several options for grid swap policy. Some of them have odd side >> effects. >> > It's better to explicitly write them down to select the best solution. >> > >> > 1. Strict policy. >> > - @IgniteSpiConsistencyChecked(optional = false) used in >> FileSwapSpaceSpi >> > and NoopSwapSpaceSpi. Nodes with different swap providers could not >> coexist >> > in the grid. >> > - Cache with swap space could be start, only if all nodes in the grid >> has >> > FileSwapSpaceSpi provider enabled. >> > In this case if part of nodes in the grid use FileSwapSpaceSpi and other >> > part use NoopSwapSpaceSpi, than only random part of them will start. If >> the >> > first started node has FileSwapSpaceSpi, then all nodes with >> > NoopSwapSpaceSpi will fail to start, and vice versa. >> > Interesting thing is that client nodes ( with >> Ignition.setClientMode(true)) >> > should also has the same swap space provider as others nodes in the >> grid. >> > I.e. If nodes in the grid use FileSwapSpaceSpi, client will fail to >> start >> > if it use NoopSwapSpaceSpi. That is strange, because client node does >> not >> > store any data, so swap policy is irrelevant to them. >> > Another interesting thing with client nodes is that client node with >> > NoopSwapSpaceSpi will never be able to start cache with swap enabled: >> > - if all nodes in the grid use FileSwapSpaceSpi, than client node simply >> > will fail to start >> > - if all nodes in the grid use NoopSwapSpaceSpi, than cache with swap >> will >> > not start on such nodes >> > 2. Soft policy >> > - @IgniteSpiConsistencyChecked(optional = true) used in FileSwapSpaceSpi >> > and NoopSwapSpaceSpi. Nodes with different swap providers could coexist, >> > warning is printed in case of different providers. >> > - Cache with swap space could be start, only if all nodes in the grid >> has >> > FileSwapSpaceSpi provider enabled. >> > In this case success of dynamic cache with swap creation depends on >> order >> > in which the nodes start. Suppose we have node with swap (nSwap), node >> > without swap (nNoSwap) and a client, who just after start creates >> dynamic >> > cache with swap (cSwap). >> > If nSwap and nNoSwap started before cSwap, than cSwap will fail to >> create >> > cache, because not all nodes in the grid support swap space. >> > If nSwap started first, than started cSwap. cSwap will create cache >> > succesfuly. After that nNoSwap will start. And we have configuration >> now, >> > when nodes with different swap policies are coexist in grid, but cache >> with >> > swap successfully started. >> > 3. Another option, initially suggested by Artem, is always use >> > FileSwapSpaceSpi if user doesn't provide SwapSpaceSpi explicitly. >> Probably >> > will get higher resources usage in this case. >> > 4. Leave it as now, but with logging: >> > - @IgniteSpiConsistencyChecked(optional = true) >> > - log when cache with swap enabled started and there some nodes without >> > swap in the grid >> > Which option should we choose? >> > >> > >> > On Tue, Feb 16, 2016 at 2:55 PM, Yakov Zhdanov >> > wrote: >> > >> > > I think, yes >> > > >> > > --Yakov >> > > >> > > 2016-02-16 14:19 GMT+03:00 Konstantin Margorin : >> > > >> > > > Should we use optional=false in @IgniteSpiConsistencyChecked ? >> > > > >> > > > On Tue, Feb 16, 2016 at 1:15 PM, Dmitriy Setrakyan < >> > > dsetrak...@apache.org> >> > > > wrote: >> > > > >> > > > > Sounds good to me. >> > > > > >> > > > > On Tue, Feb 16, 2016 at 2:13 AM, Yakov Zhdanov < >> yzhda...@apache.org> >> > > > > wrote: >> > > > > >> > > > > > Let's require that all nodes should have same swap space SPI. I >> > think >> > > > we >> > > > > > can put @GridSpiConsistencyChecked on file swap space SPI (any >> > other >> > > > > > suggestions?). >> > > > > > >> > > > > > --Yakov >> > > > > > >> > > > > > 2016-02-15 23:31 GMT+03:00 Konstantin Margorin < >> rus...@gmail.com>: >> > > > > > >> > > > > > > Probably I can handle this ticket, but I want to clarify one >> > thing. >> > > > > > > >> > > > > > > Suppose dynamic cache with enabled swap should be partitioned >> > > between >> > > > > two >> > > > > > > nodes with backups=1. First node started with >> FileSwapSpaceSpi, >> > > > second >> > > > > > wit
[GitHub] ignite pull request: IGNITE-2654
GitHub user ilantukh opened a pull request: https://github.com/apache/ignite/pull/504 IGNITE-2654 Just for demonstration, do not merge. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ilantukh/ignite ignite-2654 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/504.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #504 commit 6805ee650412baf6346d9d7cc3aa3a2a09f9bc01 Author: Ilya Lantukh Date: 2016-02-02T12:27:17Z GridNearLockRequestV2 example implementation. commit 466528329942f009c9e591a16b9453b95830b0cb Author: vozerov-gridgain Date: 2016-02-03T07:24:01Z IGNITE-2532: WIP. Only refactorings for now. commit 29c2aee6b6ffac4f27da3732575f7d82b0e8bedd Author: vozerov-gridgain Date: 2016-02-03T08:09:13Z IGNITE-2532: WIP. Only refactorings for now. commit 2a1a31d2d0999fdb8854a05fd7a75a4cd0b159a4 Author: vozerov-gridgain Date: 2016-02-03T08:36:39Z IGNITE-2532: Single update request is finalyl wired up. Though, it is not optimzied yet. commit 52d20cdcdc886c6ceaed49239e822a1d6deaa7dd Author: vozerov-gridgain Date: 2016-02-03T09:03:31Z IGNITE-2532: WIP on single message optimization. commit 89c8074452b4bc209eb03d758e534f0ef8365d46 Author: vozerov-gridgain Date: 2016-02-03T09:08:18Z IGNITE-2532: WIP on single message optimization. commit 07c23931f9758497db50bf0851af5d6c0fb8eaa4 Author: vozerov-gridgain Date: 2016-02-03T09:45:42Z IGNITE-2532: Reverting changes to GridNearAtomicUpdateFuture. commit e066650cf4907851dca1d8ef0297f6a1522d6a87 Author: vozerov-gridgain Date: 2016-02-03T12:02:52Z Merge branch 'master' into ignite-2523 commit 3967130f54fa21d25e7e284ecabaf004b937b921 Author: vozerov-gridgain Date: 2016-02-03T12:53:07Z IGNITE-2523: Finalization. commit 3d34e50c9ece0d12e35a0ff6130659cfa03bdddf Author: vozerov-gridgain Date: 2016-02-04T07:44:33Z Merge branch 'master' into ignite-2523 commit d3e1645aaeb8fd9393744cb087a9ddde27f2e95b Author: vozerov-gridgain Date: 2016-02-04T07:52:48Z IGNITE-2523: Fixed message count tests. commit 0128f988c60cdbd9aec6996626b6ffb1ad1fa30e Author: vozerov-gridgain Date: 2016-02-04T07:54:25Z IGNITE-2523: Fixed IgniteCacheAtomicStopBusySelfTest. commit 7e09a146acefccc885e78c57ca85f754d4db5d45 Author: vozerov-gridgain Date: 2016-02-04T07:59:59Z IGNITE-2523: Fixed several other tests. commit c54873bba2b2a0777a7ade33f5ec3da82c96ce95 Author: vozerov-gridgain Date: 2016-02-04T08:04:58Z IGNITE-2523: Renaming: interface -> base. commit e83408060970aba8f3f98ec7be1d96bab4fa888c Author: vozerov-gridgain Date: 2016-02-04T09:05:56Z IGNITE-2523: Fixing tests. commit eaeec214588ced814fb0ef0ef78393af90932c95 Author: Ilya Lantukh Date: 2016-02-04T15:53:26Z ignite-2533 : GridNearLockFuture redesign to optimize memory allocation. commit 1491c1f494e47618191ae7e4f79c67a5fdd9a326 Author: vozerov-gridgain Date: 2016-02-05T07:04:04Z IGNITE-2523: Renamings. commit 11a27f74f5a85320f6473a4cd2e59e35459cc587 Author: vozerov-gridgain Date: 2016-02-05T07:07:02Z Merge branch 'master' into ignite-2523 commit a18c352748a86ac9e48885f1b6ae93ff9ed4f4dd Author: vozerov-gridgain Date: 2016-02-05T07:09:48Z IGNITE-2523: Minors. commit 649d7cd3dc87f09c9fa0c4476bbc65ab2728824d Author: Ilya Lantukh Date: 2016-02-05T10:47:15Z ignite-2533 : Fixed deadlock on topology error. commit dfdd2f4f907651a86da6fd52f18d2f189f65279d Author: Ilya Lantukh Date: 2016-02-08T14:23:02Z ignite-2523 : Created SingleUpdateResponse commit c39410a2dbbab7e8886a407a5661b29ee985adce Author: Ilya Lantukh Date: 2016-02-08T15:05:34Z ignite-2523 : SingleUpdateResponse implementation. commit 3c8d02a00b2bcbc194fd2112d9f8cb58ab7d571a Author: Ilya Lantukh Date: 2016-02-08T15:25:21Z ignite-2523 : Generalized usage of GridNearAtomicUpdateRequest/Response. commit cb5bdb3f19cefe14380ac169d49e6e86bde1899a Author: Ilya Lantukh Date: 2016-02-08T15:51:55Z ignite-2523 : Generalized usage of GridNearAtomicUpdateRequest/Response. commit 4f8220e6da06443f1ef42ac3f4b4e46f5100dcd1 Author: Ilya Lantukh Date: 2016-02-09T11:38:12Z ignite-2523 : finished generalization of GridNearAtomicUpdate. commit 2f6aff8024794cd980fd77bb96e9a74876fb7442 Author: Ilya Lantukh Date: 2016-02-09T12:04:52Z ignite-2523 : Fixed failing tests. commit 6040f45eae844fd40c113de452e70078dae9a95c Author: Ilya Lantukh Date: 2016-02-09T13:14:57Z ignite-2523 : Fixed test failures. commit 3391c847e9825405d8ca2f7df7e01ba432f60b1b Author: Ilya Lantukh Date: 2016-02-09T14:43:39Z ignite-2523 : Created GridDhtAtomicSingleUpdateRequest optimized implementation. commit cd07298f31be580b8341e0409b4669376e90f1bd Author: Ilya Lantukh Date: