Re: Spark Data Frame support in Ignite

These are two different things. Spark applications themselves do not use JDBC - it is more for non-spark applications to access Spark DataFrames. A direct support by Ignite would make more sense. Although you have in theory IGFS, if the user is using HDFS, which might not be the case. It is now

Spark Data Frame support in Ignite

Igniters, We have had the integration with Spark Data Frames on our roadmap for a while: https://issues.apache.org/jira/browse/IGNITE-3084 However, while browsing Spark documentation, I cam across the generic JDBC data frame support in Spark:

Re: Cluster auto activation design proposal

First of all, this sounds a bit too scientific and will likely be misunderstood and misused. Is there a chance to give it a better name and a simpler use case and description? Secondly, I do agree that it should be mandatory to update the BT. For example, what if I want to add several nodes to

Re: Thin client protocol message format

Let us not forget that the main purpose of such protocol is to enable other users contribute their own client implementations for various languages. Also, most JDBC and ODBC use cases work in thread-per-connection mode. I think that if we introduce a multi-threaded client here, then it will be a

Re: Create own SQL parser

Vladimir, this sounds like a task that would take years to design, implement, and polish. Can we just aim to improve the Ignite mode in H2, which is much more feasible in my view? On Wed, Aug 2, 2017 at 2:59 PM, Vladimir Ozerov wrote: > Alex P., > > The very problem with

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Here is what we should do: 1. Pick an acceptable number. Does not matter if it is 10% or 50%. 2. Print the allocated memory in *BOLD* letters into the log. 3. Make sure that Ignite server never hangs due to the low memory issue. We should sense it and kick the node out automatically,

Re: Release notes tools for Ignite releases

Agree with Denis. I think that we should treat it as a starting point for the release notes, and further update it before releases. On Wed, Aug 2, 2017 at 10:58 PM, Denis Magda wrote: > Guys, > > We're on the same page that the page has to look more colorful and >

Re: Release notes tools for Ignite releases

Guys, We're on the same page that the page has to look more colorful and informative. This is exactly why Spark's page was given as an example. However, I like the template generated by Alex script in a sense that you already have something to start work with. Go ahead and remove redundant

Re: Release notes tools for Ignite releases

Alex, This is not only about the issue types. Release notes is the face of our product. Generating it from JIRA has several problems which are unresolvable IMO: 1) Tickets are created by many dozens people, so there could be typos, linguistic errors, etc. 2) Ticket descriptions often inconclusive

Re: Release notes tools for Ignite releases

Vladimir, I agree that page not perfect now. I think, problem in Jira issue types. A lot of new features or improvements created as task and often we don't use sub-task for depended issues. I can be wrong, but committer who prepared release can't manually create this page, we need have some

Re: Set default cache synchronization mode to FULL_SYNC

We have to wait with any default changes to 3.0, unfortunately. On Wed, Aug 2, 2017 at 8:30 PM, Vladimir Ozerov wrote: > Not sure about readFromBackup, but changing PRIMARY_SYNC to FULL_SYNC looks > safe to me. Any other thoughts? > > ср, 2 авг. 2017 г. в 21:10, Denis

Re: Release notes tools for Ignite releases

JIRA = report from JIRA ср, 2 авг. 2017 г. в 21:35, Vladimir Ozerov : > Denis, > > This page works exactly how I suggested in the beginning of that thread: > manually crafted notes on most important features + link to JIRA report to > see all closed tickets. > > We already

Re: Release notes tools for Ignite releases

Denis, This page works exactly how I suggested in the beginning of that thread: manually crafted notes on most important features + link to JIRA report to see all closed tickets. We already have manually crafted any properly grouped release notes. All we need is to make them a bit more verbose,

Re: Set default cache synchronization mode to FULL_SYNC

Not sure about readFromBackup, but changing PRIMARY_SYNC to FULL_SYNC looks safe to me. Any other thoughts? ср, 2 авг. 2017 г. в 21:10, Denis Magda : > +1 for both suggestions but I’m not sure we can do the change till 3.0. > > — > Denis > > > On Aug 2, 2017, at 1:27 AM,

Re: Release notes tools for Ignite releases

Vladimir, The goal is to have a page like that: https://spark.apache.org/releases/spark-release-2-1-0.html where a user can go and see all the changes incorporated in the release. The header of the file can be custom - you can list

Re: Set default cache synchronization mode to FULL_SYNC

+1 for both suggestions but I’m not sure we can do the change till 3.0. — Denis > On Aug 2, 2017, at 1:27 AM, Vladimir Ozerov wrote: > > +1 for readFromBackup=false as well :-) Another example of default value > with subtle effects. > > On Wed, Aug 2, 2017 at 11:11 AM,

[GitHub] ignite pull request #2373: IGNITE-5759: cache 5 suite timed out by GridCache...

Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/2373 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: Add isPrimary() and isBackup() methods on CacheQueryEntryEvent

Sure. CacheQueryEntryEvent: public abstract boolean isBackup(); public abstract boolean isPrimary(); > 2 авг. 2017 г., в 19:56, Nikolai Tikhonov написал(а): > > Max, > > Thank you for your contribution! Could you share here what exactly was > added to interface? > >

Re: SQL Getting Started Guide

Akmal, The contributor permissions are granted in JIRA. Please go ahead and assign the ticket on yourself. Keep us posted on the progress and don’t hesitate raising any questions if you face troubles. — Denis > On Aug 2, 2017, at 7:17 AM, Akmal Chaudhri >

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Vladimir, Dmitriy P., Please see inline > On Aug 2, 2017, at 7:20 AM, Vladimir Ozerov wrote: > > Denis, > > The reason is that product should not hang user's computer. How else this > could be explained? I am developer. I start Ignite, 1 node, 2 nodes, X > nodes, observe

Re: Release notes tools for Ignite releases

Alex, In AI 2.1 we fixed several hundreds issues. Why do you think there is a single person interested in reviewing all of them? E.g. we added new JDBC driver. I do not see it in the list of major features, neither I need to know that this task was split into 20 smaller sub tasks, each of which

Re: Release notes tools for Ignite releases

Vladimir, We have links to release notes on download page: https://ignite.apache.org/ download.cgi. For be user-friendly we added header with description about most interested features in release and near list of issues. If we want skip minor or another issues we can use Jira labels and skip

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Denis, The reason is that product should not hang user's computer. How else this could be explained? I am developer. I start Ignite, 1 node, 2 nodes, X nodes, observe how they join topology. Add one key, 10 keys, 1M keys. Then I do a bug in example and load 100M keys accidentally - restart the

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Hi Igniters, When I was Ignite user before installing product on production server I’ve always used this page https://apacheignite.readme.io/docs/jvm-and-system-tuning for selecting appropriate parameters. But before go live, I’ve used Ignition.start() 20 times per day to check if my changes in

Re: SQL Getting Started Guide

Denis, All: Jira account details: Username: abchaudhri Email: akmal.chaudhri AT gridgain.com Full Name: Akmal Chaudhri Thank you. On 31 July 2017 at 19:02, Denis Magda wrote: > Igniters, > > There is a lot of SQL related documentation available for Ignite. However, > the

Re: Release notes tools for Ignite releases

Alex. How are we supposed to use this report? Release notes should be designed in such a way, that the most important features are highly visible, while minor changes are either skipped at all, or shown aside, as they have little to no value for users. If someone is interested in all fixed

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Sergey, That’s expectable because as we revealed from this discussion the allocation works different depending on whether the persistence is used or not: 1) In-memory mode (the persistence is disabled) - the space will be allocated incrementally until the max threshold is reached. Good! 2)

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Denis, Just a simple example from our own codebase: I tried to execute PersistentStoreExample with default settings and two server nodes and client node got frozen even on initial load of data into the grid. Although with one server node the example finishes pretty quickly. And my laptop isn't

Re: [IGNITE-5717] improvements of MemoryPolicy default size

> As far as allocating 80% of available RAM - I was against this even for > In-memory mode and still think that this is a wrong default. Looking at > free RAM is even worse because it gives you undefined behavior. Guys, I can not understand how this dynamic memory allocation's high-level

Release notes tools for Ignite releases

Igniters, Started 2.0 we generated manually Jira based release notes with all new improvements and fixed bugs. f.e(https://ignite.apache.org/releases/2.1.0/release_notes.html). I created new class in ignite-tools for we can do it automatically using IDE or TeamCity. Class using JSON template for

[GitHub] ignite pull request #2380: Ignite 2.1.4

GitHub user AMashenkov opened a pull request: https://github.com/apache/ignite/pull/2380 Ignite 2.1.4 You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-2.1.4 Alternatively you can review and apply

Re: Create own SQL parser

Alex P., The very problem with non-SELECT and non-DML commands is that we do not support most of what is supported by H2, and vice versa - H2 doesn't support most things that we need, like cache properties, templates, inline indexes, etc.. Another important thing is that at some point we will add

Re: Data compression in Ignite 2.0

Vova, Finally we back to my initial idea - to look how "big databases compress" data :) Just to remind how IBM DB2 do this[1]. [1] http://www.ibm.com/developerworks/data/library/techarticle/dm- 1205db210compression/ On Tue, Aug 1, 2017 at 4:15 PM, Vladimir Ozerov wrote:

Re: Create own SQL parser

>From my opinion we could start investing in our own parser and SQL engine step by step, little by little and one day drop H2 at all. Having own parser and engine will give us a freedom to any optimizations and any syntax we like. Also that will be for one dependency less and we could have SQL

[GitHub] ignite pull request #2379: IGNITE-5355 Implement class for generate Jira bas...

GitHub user achetaev opened a pull request: https://github.com/apache/ignite/pull/2379 IGNITE-5355 Implement class for generate Jira based release report You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite

[GitHub] ignite pull request #2089: IGNITE-5355 Implement class for generate Jira bas...

Github user achetaev closed the pull request at: https://github.com/apache/ignite/pull/2089 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] ignite pull request #2378: Ignite-5829 Link DataRecord with DataPage records

GitHub user Jokser opened a pull request: https://github.com/apache/ignite/pull/2378 Ignite-5829 Link DataRecord with DataPage records You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-5829

Re: Thin client protocol message format

Yakov, >This should be set up in handshake and used for all requests made over that connection. Sounds reasonable. On Wed, Aug 2, 2017 at 6:29 PM, Yakov Zhdanov wrote: > > Response should contains Node ID that send response to thin client. > > This should be set up in

Re: Thin client protocol message format

> Response should contains Node ID that send response to thin client. This should be set up in handshake and used for all requests made over that connection. --Yakov 2017-08-02 14:12 GMT+03:00 Alexey Kuznetsov : > Pavel, > > I remember one thing that is very useful for

[jira] [Created] (IGNITE-5905) .NET: Thin client: cache.Get for primitives

Pavel Tupitsyn created IGNITE-5905: -- Summary: .NET: Thin client: cache.Get for primitives Key: IGNITE-5905 URL: https://issues.apache.org/jira/browse/IGNITE-5905 Project: Ignite Issue Type:

[GitHub] ignite pull request #2376: IGNITE-5899 Thin client: cache.Get for primitives

GitHub user ptupitsyn opened a pull request: https://github.com/apache/ignite/pull/2376 IGNITE-5899 Thin client: cache.Get for primitives You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-5899

[GitHub] ignite pull request #2334: IGNITE-5712: implementation of context switching ...

Github user nizhikov closed the pull request at: https://github.com/apache/ignite/pull/2334 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: Thin client protocol message format

Pavel, 2. Disagree here. In your sample responses contain only success flag which is byte and 4 bytes length. We can easily avoid that just sending OP_CODE_SUCCESS or OP_CODE_FAILURE. 3. In and out where applicable. --Yakov

[jira] [Created] (IGNITE-5904) .NET: Improve array access in IBinaryReader and IBinaryRawReader

Pavel Tupitsyn created IGNITE-5904: -- Summary: .NET: Improve array access in IBinaryReader and IBinaryRawReader Key: IGNITE-5904 URL: https://issues.apache.org/jira/browse/IGNITE-5904 Project: Ignite

Re: Thin client protocol message format

Pavel, I do not see how it can effect something existing, as nothing exists yet :-) Let's put current SQL aside, we will merge them into protocol later. The main point of length is to move request parsing and deserialization to separate thread. Without it we will have to pre-process all requests

Re: Thin client protocol message format

Yakov, > 2. put op_code to the first place. This will make possible to eliminate length field for many messages This will require extensive refactoring of existing socket pipeline for very little benefit: almost all messages are of variable length. > 3. build date and revision hash to handshake

Re: Thin client protocol message format

Agree with Alex. I think our implementations should share single connection over threads in the process. --Yakov

[jira] [Created] (IGNITE-5903) .NET: Add tests for CREATE/DROP table/index

Pavel Tupitsyn created IGNITE-5903: -- Summary: .NET: Add tests for CREATE/DROP table/index Key: IGNITE-5903 URL: https://issues.apache.org/jira/browse/IGNITE-5903 Project: Ignite Issue Type:

Re: Thin client protocol message format

I think that in the first iteration it would be enough to have connection-per-thread approach. But in future we definitely would like to multiplex threads over a single connection and to support async operations. That said, we definitely need request ID. Let's add it right now to avoid

Re: Set default cache synchronization mode to FULL_SYNC

+1 for readFromBackup=false as well :-) Another example of default value with subtle effects. On Wed, Aug 2, 2017 at 11:11 AM, Alexey Goncharuk < alexey.goncha...@gmail.com> wrote: > Vladimir, > > Personally, I agree that we should put correctness over performance, > however (1) is not a correct

[jira] [Created] (IGNITE-5902) Refactor VisorCacheStopTask to support stopping several caches at once

Alexey Kuznetsov created IGNITE-5902: Summary: Refactor VisorCacheStopTask to support stopping several caches at once Key: IGNITE-5902 URL: https://issues.apache.org/jira/browse/IGNITE-5902

[GitHub] ignite pull request #2375: Ignite gg 5038 benchmark

GitHub user vldpyatkov opened a pull request: https://github.com/apache/ignite/pull/2375 Ignite gg 5038 benchmark You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-gg-5038_benchmark Alternatively

Re: Set default cache synchronization mode to FULL_SYNC

Vladimir, Personally, I agree that we should put correctness over performance, however (1) is not a correct statement for TRANSACTIONAL caches. A transactional client always validates the result of an operation and throw a correct exception if operation failed. (1) is true for ATOMIC caches,

Re: Thin client protocol message format

Do I understand correctly that this is not a multiplexed protocol? Are we ok to have a separate connection for each thread? I would also add a requestId field to allow multiple concurrent requests at a time. 2017-08-02 10:50 GMT+03:00 Vladimir Ozerov : > Yakov, > > Yes,

Re: Cluster auto activation design proposal

I think we should be able to change the BT in the runtime, and a user should have several ways to do this: * programmatically via the API suggested by Sergey * Using management tools (console visor on Web Console) * Based on some sort of policies when the actual cluster topology differs too

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Free RAM is variable. For example, I can start Ignite before IDE (Eclipse, or so), or after IDE. In the first case it will hang, in the send it won't. Unstable. To be clear - I propose to have "maxMemory" set to 10%, not "initialMemory". It doesn't matter how exactly we reach it - in one hop, or

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Dmitriy, The reason behind this is the need to to be able to evict and load pages to disk, thus we need to preserve a PageId->Pointer mapping in memory. In order to do this in the most efficient way, we need to know in advance all the address ranges we work with. We can add dynamic memory

Re: Thin client protocol message format

Here are my observations. 1. Let's create wiki page where we will keep the protocol definition and reflect all the changes. 2. I would put op_code to the first place. This will make possible to eliminate length field for many messages. Look at your handshake request - it is always of fixed

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Dima, Probably folks who worked closely with storage know why. However, even increments will not help us. Most developers work on laptops. Common laptop has 8-16 Gb RAM today. OS and working programs typically consume about a half. So, for example, I have 8Gb total, 4Gb consumed, 4Gb free. And

Re: [IGNITE-5717] improvements of MemoryPolicy default size

On Wed, Aug 2, 2017 at 7:27 AM, Vladimir Ozerov wrote: > Please see original Sergey's message - when persistence is enabled, memory > is not allocated incrementally, maxSize is used. > Why? > Default settings must allow for normal work on developer's environment. >

[GitHub] ignite pull request #2374: Ignite 5860 Client disconnects if server it is co...

GitHub user dmekhanikov opened a pull request: https://github.com/apache/ignite/pull/2374 Ignite 5860 Client disconnects if server it is connected to goes unresponsive You can merge this pull request into a Git repository by running: $ git pull

Re: Create own SQL parser

No, it will work as follows: Model parse(String sql) { Model res = tryParseWithIgnite(sql); // Parse what we can if (res == null) res = parseWithH2(sql); return res; } We will need a number of custom commands which are not present in H2. On Wed, Aug 2, 2017 at 3:44 AM,