Dima,
There is no need for these extra bytes. Protocol version would be enough.
Extensibility points wil be designed separately. We already have experience
with it in scope of .NET/CPP platforms, it is solved by special command
types where user can hook his own code.
On Wed, Aug 2, 2017 at 3:56
Please see original Sergey's message - when persistence is enabled, memory
is not allocated incrementally, maxSize is used.
But this is a problem even for in-memory case. Consider that I wanted to
load several gigs of data as an experiment on my laptop - frequent
scenario. First several seconds
How about allowing 8 bytes of extra space for any user-specific additional
data? For example, I can see that some implementations of this protocol
will need to pass some extra parameters to the server. Of course, in that
case the server side protocol handler should have a hook to react to these
On Tue, Aug 1, 2017 at 11:08 PM, Vladimir Ozerov
wrote:
> Own parser capable of processing non-SELECT and non-DML statements.
>
And how will it integrate with H2 parser? Or are you suggesting that we get
rid of H2 SQL parser?
>
> On Tue, Aug 1, 2017 at 9:44 PM,
> Why not allocate in increments automatically?
This is exactly how the allocation works right now. The memory will grow
incrementally until the max size is reached (80% of RAM by default).
—
Denis
> On Aug 1, 2017, at 3:03 PM, dsetrak...@apache.org wrote:
>
> Vova, 1GB seems a bit too small
Vova, 1GB seems a bit too small for me, and frankly i do not want t o guess.
Why not allocate in increments automatically?
D.
On Aug 1, 2017, 11:03 PM, at 11:03 PM, Vladimir Ozerov
wrote:
>Denis,
>No doubts you haven't heard about it - AI 2.1 with persistence, when
Vladimir,
80% allocation approach was implemented in AI 2.0. I personally explained users
on the forums its behavior.
Actually, dynamic vs static memory allocation scenarios were analyzed and
discussed long time ago and eventually we decided to go for with the former:
Own parser capable of processing non-SELECT and non-DML statements.
On Tue, Aug 1, 2017 at 9:44 PM, wrote:
> Vova, I am not sure what you are proposing... extending H2 parser with new
> syntax or a brand new parser?
>
> D.
>
> On Aug 1, 2017, 4:26 PM, at 4:26 PM,
Exactly.
On Tue, Aug 1, 2017 at 10:16 PM, Denis Magda wrote:
> Vladimir,
>
> Do you mean a dedicated documentation tab like we have for .NET, C++,
> Integration and tools?
>
> —
> Denis
>
> > On Aug 1, 2017, at 1:14 AM, Vladimir Ozerov
> wrote:
> >
> >
Denis,
No doubts you haven't heard about it - AI 2.1 with persistence, when 80% of
RAM is allocated right away, was released several days ago. How do you
think, how many users tried it already?
Guys,
Do you really think allocating 80% of available RAM is a normal thing? Take
your laptop and check
Andrey,
What I’m trying to say is that presently both text and geo-spatial indexes are
unsupported NOT only for the persistence layer but for the durable (page)
memory in general - as you properly pointed out the text indexes are stored in
the old off-heap memory but has to be moved to be
Denis,
I see we still use GridUnsafeMemory in LuceneIndex [1] in master.
Do you mean a wrapper to new memory manager is used here?
[1]
https://github.com/apache/ignite/blob/master/modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/opt/GridLuceneIndex.java
On Tue, Aug
Andrey,
I’ve already wiped the old off-heap memory out of my mind. Sure I assumed the
off-heap space managed by the new memory architecture.
—
Denis
> On Aug 1, 2017, at 11:14 AM, Andrey Mashenkov
> wrote:
>
> Denis,
>
> Lucene fullText index stores off-heap in
My vote goes for option #1 too. I don’t think that 80% is too aggressive to
bring it down.
IGNITE-5717 was created to fix the issue of the 80% RAM allocation on 64 bit
systems when Ignite works on top of 32 bit JVM. I’ve not heard of any other
complaints in regards the default allocation size.
Good job, Vitaliy. Keep the pace!
—
Denis
> On Aug 1, 2017, at 7:53 AM, Vitaliy Biryukov
> wrote:
>
> Hi Igniters,
>
> While working on one of issuess, I noticed that the
> "IgniteUtils.ceilPow2" method can overflow.
>
> I created issue[1] and fixed this method
Sergey,
Is it assumed that the baseline topology can be updated in runtime? Like,
initially I had a cluster of 10 nodes but in a couple of weeks it was expanded
to 15 nodes. How the baseline topology should be updated in this way? Will it
happen automatically?
—
Denis
> On Aug 1, 2017, at
Vladimir,
Do you mean a dedicated documentation tab like we have for .NET, C++,
Integration and tools?
—
Denis
> On Aug 1, 2017, at 1:14 AM, Vladimir Ozerov wrote:
>
> Igniters,
>
> I think it is time to create separate section for SQL-related features. We
> already
I would prefer that we reuse an existing compression protocol, but at the table
level.
If not possible, then we should go with a shared mapping approach. Any idea how
hard?
D.
On Aug 1, 2017, 11:15 AM, at 11:15 AM, Vladimir Ozerov
wrote:
>Vyacheslav,
>
>This is not
Vova, I am not sure what you are proposing... extending H2 parser with new
syntax or a brand new parser?
D.
On Aug 1, 2017, 4:26 PM, at 4:26 PM, Vladimir Ozerov
wrote:
>Andrey,
>
>Note that I am not proposing to remove H2 as a whole. Main point for
>now is
>to support
Denis,
Lucene fullText index stores off-heap in old way, not in page memory.
Therefore it is not persistent.
There is a second issue related to FullText index as it is different kind
of index and it has own query type and it can't be use in SQL queries.
Looks like it make sense to integrate
Igniters,
The practice shows that regardless of Ignite’s project maturity we as a
community still miss some valuable points and principles on different occasions
violating the ASF processes.
Cos Boudnik as one of the oldest ASF and Ignite members agreed to conduct a
global webinar reiterating
Dmitriy Pavlov created IGNITE-5901:
--
Summary: Ignite PDS Indexing Suite:
IgniteWalRecoveryTest.testWalRolloverMultithreadedLogOnly is now flaky on TC
with AsynchronousCloseException
Key: IGNITE-5901
URL:
Yes, in my understanding, we should provide backwards compatibility via
versioning.
During handshake we pick RequestParser and RequestHandler (this is already
implemented, see SqlListenerNioListener.prepareContext).
Each protocol version would have separate implementations of these
interfaces.
Vladimir, what about binary mode (IgniteCache.withKeepBinary)?
Two caches may have different encoding settings:
BinaryObject obj = cache1.get(key); // Got fields in utf8
cache2.put(key, obj); // Fields are expected to be in Windows-1251
What do we do here? Re-build the binary object?
Also,
Backward compatible?
D.
On Aug 1, 2017, 7:04 PM, at 7:04 PM, Pavel Tupitsyn
wrote:
>Dmitry, we don't need any reserved bytes, because protocol is
>versioned.
>
>On Tue, Aug 1, 2017 at 7:49 PM, wrote:
>
>> We should also leave 8 bytes of empty
Vladimir,
We need to consider that these two types of indexes are not stored off-heap
either.
It expands the task a bit — the indexes have to be fully integrated with the
new durable memory architecture supporting both off-heap and persistence layers.
—
Denis
> On Aug 1, 2017, at 3:26 AM,
Dmitry, we don't need any reserved bytes, because protocol is versioned.
On Tue, Aug 1, 2017 at 7:49 PM, wrote:
> We should also leave 8 bytes of empty space for future changes.
>
> D.
>
> On Aug 1, 2017, 6:41 PM, at 6:41 PM, Pavel Tupitsyn
>
We should also leave 8 bytes of empty space for future changes.
D.
On Aug 1, 2017, 6:41 PM, at 6:41 PM, Pavel Tupitsyn
wrote:
>Alexey, good idea. ODBC and JDBC could also benefit from this.
>
>On Tue, Aug 1, 2017 at 7:27 PM, Alexey Kuznetsov
>
Igor Sapego created IGNITE-5900:
---
Summary: CPP: Add tests for CREATE/DROP table/index for C++ and
ODBC
Key: IGNITE-5900
URL: https://issues.apache.org/jira/browse/IGNITE-5900
Project: Ignite
Alexey, good idea. ODBC and JDBC could also benefit from this.
On Tue, Aug 1, 2017 at 7:27 PM, Alexey Kuznetsov
wrote:
> Pavel,
>
> How about data compression?
> May be it make sense to add a byte with compression algorithm?
> 0 - none
> 1 - ZIP
> 2 -
>
>
> On
For supporting different cases, might be useful to add to handshake message
one byte for different flags. compression is a good example. Also it can be
some specific logic for different clients (C++, JS, Python), crypting and
etc.
On Tue, Aug 1, 2017 at 7:27 PM, Alexey Kuznetsov
Pavel,
How about data compression?
May be it make sense to add a byte with compression algorithm?
0 - none
1 - ZIP
2 -
On Tue, Aug 1, 2017 at 11:10 PM, Pavel Tupitsyn
wrote:
> Igniters,
>
> Below is a proposed design for thin client protocol [1] [2] socket data
Hi, Nikolay,
I guess it may be related to
http://apache-ignite-developers.2346864.n4.nabble.com/usage-analytics-td19504.html
Test is now decided to be muted until decision is made.
Actual scope of failures can be found in
Hello, Igniters.
`GridVersionSelfTest.testVersions` fails for a long time:
http://ci.ignite.apache.org/project.html?tab=testDetails=Ignite20Tests=2028128591644736718=10
In current master it muted by IGNITE-5413. But IGNITE-5413 is resolved now.
Test muted by Sergey Chugunov in
Igniters,
Below is a proposed design for thin client protocol [1] [2] socket data
exchange format.
* Values are little-endian
* Every request and response message starts with 4-byte length (including
handshake)
* Ignite binary format is used for value serialization (via
GitHub user dspavlov opened a pull request:
https://github.com/apache/ignite/pull/2373
IGNITE-5759: cache 5 suite timed out by
GridCachePartitionEvictionDuringReadThroughSelfTest.testPartitionRent
More debug info, cancelling threads in case of timeout: cache 5 suite timed
out by
GitHub user nizhikov opened a pull request:
https://github.com/apache/ignite/pull/2372
IGNITE-425: Implementation of ContinuousQueryWithTransformer
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nizhikov/ignite IGNITE-425
Igniters,
I've started the work on thin client protocol [1].
We are going to reuse existing socket pipeline from ODBC/JDBC
by adding one more client type to SqlListenerNioListener
(see prepareContext in there to get an idea of how this works).
First iteration will be stateless and synchronous,
Hi Igniters,
While working on one of issuess, I noticed that the
"IgniteUtils.ceilPow2" method can overflow.
I created issue[1] and fixed this method retaining its bitHack style.
This fix was merged to the master branch, thanks to Anton Vinogradov .
[1]
Hello Ignite developers,
I would like to start a discussion about design of important improvement
enabling automatic activation of cluster with durable store turned on [1].
Also it will help us to solve an issue with data divergence (e.g. this may
happen when half of the cluster goes down and
Pavel Tupitsyn created IGNITE-5899:
--
Summary: Thin client: cache.Get for primitives
Key: IGNITE-5899
URL: https://issues.apache.org/jira/browse/IGNITE-5899
Project: Ignite
Issue Type: Task
Andrey,
Note that I am not proposing to remove H2 as a whole. Main point for now is
to support missing pieces of DDL syntax and possibly and some extensions.
Several examples:
1) Currently CREATE TABLE command looks ugly:
CREATE TABLE Person (id LONG, name VARCHAR) WITH "template=PARTITIONED,
GitHub user tledkov-gridgain opened a pull request:
https://github.com/apache/ignite/pull/2371
IGNITE-5211 Classes based constructor for QueryEntities
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gridgain/apache-ignite
Hi Igniters,
There was warning of building of the core module[1].
The reason was guava-dependency duplication in pom.xml
I removed outdated dependency and warning was gone.
I checked compilation with clean local repo, it passed well.
My fix was merged to the master branch, thanks to Anton
Irina Zaporozhtseva created IGNITE-5898:
---
Summary: Datagrid.QueryDmlExample: Incorrect output if run example
with standalone Apache Ignite.NET node
Key: IGNITE-5898
URL:
GitHub user mcherkasov opened a pull request:
https://github.com/apache/ignite/pull/2370
Ignite 2.1.1 p1
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gridgain/apache-ignite ignite-2.1.1-p1
Alternatively you can review and
Dmitriy Pavlov created IGNITE-5897:
--
Summary: Ignite Cache Full API Multi JVM: 7 test failed in master
Key: IGNITE-5897
URL: https://issues.apache.org/jira/browse/IGNITE-5897
Project: Ignite
Vovan,
1. What about ANSI-xx compliant? Will new syntax brake it in some cases or
just extend?
2. This would be great to have more ways for optimization.
Does anyone know or may be have experience with some frameworks or open
projects which can be helpful? E.g. Apache Calcite?
On Tue, Aug 1,
Great start! Thanks!
2017-08-01 14:54 GMT+03:00 Николай Ижиков :
> Hello, Igniters.
>
> Alexey Goncharuk just merged my fix of IGNITE-5866 to master.
>
> I fix filter in MetadataJob for `cache.context().queries().sqlMetadata()`
> function
>
> It means that 4 other tests
Vyacheslav,
I've merged changes to the master branch, thanks.
On Tue, Aug 1, 2017 at 11:54 AM, Vyacheslav Daradur
wrote:
> Hi Igniters.
>
> I found an issue wich may affect on builing of the core module.
>
> I created a ticket[1] and prepared PR[2] with TC-tests[3]
>
>
Pavel Tupitsyn created IGNITE-5896:
--
Summary: Thin client protocol
Key: IGNITE-5896
URL: https://issues.apache.org/jira/browse/IGNITE-5896
Project: Ignite
Issue Type: New Feature
Igniters,
As you know, we rely on H2 for SQL query parsing. This has several
drawbacks:
1) Limited and ugly syntax
Ignite has lot's of unique concepts which are in no way supported by
traditional RDBMS in general, or by H2 in particular. For example:
- query hints ("distributedJoins",
Kestutis Gedminas created IGNITE-5895:
-
Summary: Add support for javax.persistence.Transient
Key: IGNITE-5895
URL: https://issues.apache.org/jira/browse/IGNITE-5895
Project: Ignite
Issue
Artem Malykh created IGNITE-5894:
Summary: Implement Gini impurity calculator for column-based
decision trees.
Key: IGNITE-5894
URL: https://issues.apache.org/jira/browse/IGNITE-5894
Project: Ignite
Artem Malykh created IGNITE-5893:
Summary: Change column decision tree stopping criterion
Key: IGNITE-5893
URL: https://issues.apache.org/jira/browse/IGNITE-5893
Project: Ignite
Issue Type:
Artem Malykh created IGNITE-5892:
Summary: Check for optimal way of working with sparse data in
decision trees
Key: IGNITE-5892
URL: https://issues.apache.org/jira/browse/IGNITE-5892
Project: Ignite
Hello, Igniters.
Alexey Goncharuk just merged my fix of IGNITE-5866 to master.
I fix filter in MetadataJob for `cache.context().queries().sqlMetadata()`
function
It means that 4 other tests become green:
JettyRestProcessorSignedSelfTest#testMetatdataLocal
Github user asfgit closed the pull request at:
https://github.com/apache/ignite/pull/2367
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Vladimir Ozerov created IGNITE-5891:
---
Summary: Persistence for geo-spatial indexes
Key: IGNITE-5891
URL: https://issues.apache.org/jira/browse/IGNITE-5891
Project: Ignite
Issue Type: Task
Guys,
AFAIK these two index types are not supported with enabled persistence at
the moment, neither they stored on the disk anyhow. Can someone help with
estimates on how difficult would it be to implement these indexes over
page-memory architecture?
Looks like we will have to write our own
As a result:
We found that everything is fine and process has not been violated.
On Mon, Jul 31, 2017 at 11:09 PM, Anton Vinogradov wrote:
> Cos,
>
> 1) RC3 and RC4 files are the same because source was not changed from RC3
> to RC4.
> RC4 was upload to dev at revision 20551,
+1 to Sergey,
I think OOME may confuse user and and may spoil the user's impression at
first check of Apache Ignite.
вт, 1 авг. 2017 г. в 12:36, Vladimir Ozerov :
> +1
>
> 80% of RAM is way too aggressive. With this value virtually every user will
> have problems with OOME
Valya,
It makes sense to have both Externalizable and Binarylizable, as user might
want to serialize object for different systems. E.g. deserialize binary
stream from Kafka in Externalizable mode, and then put it to Ignite with
Binarylizable to allow for field access without deserialization.
Alexey Goncharuk created IGNITE-5890:
Summary: Add estimated time to rebalance completion and time to
rebalance start to MXBean
Key: IGNITE-5890
URL: https://issues.apache.org/jira/browse/IGNITE-5890
+1
80% of RAM is way too aggressive. With this value virtually every user will
have problems with OOME or excessive swapping during development. I would
set maxMemory to some relatively small value, may be even less that 1Gb to
let Ignite run smoothly on developer's laptops, and print a
Github user asfgit closed the pull request at:
https://github.com/apache/ignite/pull/2254
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user asfgit closed the pull request at:
https://github.com/apache/ignite/pull/1599
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user asfgit closed the pull request at:
https://github.com/apache/ignite/pull/2312
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user asfgit closed the pull request at:
https://github.com/apache/ignite/pull/2294
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Managing encoding on per-cache level is not that complex thing.
Essentially, when any cache message are prepared on initiating node, we
perform Object -> BinaryObject transition. These places has reference to
cache context ([1], [2]). This is where we should define proper string
encoding - either
Folks,
I would like to get back to the question about MemoryPolicy maxMemory
defaults.
Although MemoryPolicy may be configured with initial and maxMemory
settings, when persistence is used MemoryPolicy always allocates maxMemory
size for performance reasons.
As default size of maxMemory is 80%
Vyacheslav,
This is not about my needs, but about the product :-) BinaryObject is a
central entity used for both data transfer and data storage. This is both
good and bad at the same time.
Good thing is that as we optimize binary protocol, we improve both network
and storage performance at the
Hi Igniters.
I found an issue wich may affect on builing of the core module.
I created a ticket[1] and prepared PR[2] with TC-tests[3]
Could someone review it?
[1] https://issues.apache.org/jira/browse/IGNITE-5882
[2] https://github.com/apache/ignite/pull/2366/files
[3]
Vladislav Pyatkov created IGNITE-5889:
-
Summary: Exception message confused when double cache
configuration by some mistake.
Key: IGNITE-5889
URL: https://issues.apache.org/jira/browse/IGNITE-5889
Pavel,
"deprecate" != "delete" or "brake". We are not going to wait for 3.0. New
SQL should be designed carefully taking as mush current and future features
in count as possible. That said, in order to add new concepts like
"constraint" on PAI level we should at least start separate discussion
Igniters,
I think it is time to create separate section for SQL-related features. We
already have SQL itself, ODBC driver, several JDBC drivers, integration
with Pentaho and Tableau. Moreover, as we are going to position the product
as database, we need to have much more docs with syntax, ANSI
Pavel Konstantinov created IGNITE-5888:
--
Summary: Web console: Incorrect dependency 'undefined' with value
'null'
Key: IGNITE-5888
URL: https://issues.apache.org/jira/browse/IGNITE-5888
Project:
Denis,
This is rather simple change. Hopefully we will get it done by 2.2 release.
On Fri, May 12, 2017 at 9:16 PM, Denis Magda wrote:
> Here is a ticket:
> https://issues.apache.org/jira/browse/IGNITE-5211
>
> Vovan, can it be done in 2.1?
>
> —
> Denis
>
> > On May 11,
Denis,
I updated MyBatis integration, and will release it this week.
-- Roman
On Tuesday, August 1, 2017 5:23 AM, Denis Magda wrote:
Andrey, Roman,
As maintainers of the projects in the subject, could you please go ahead and
update Ignite version to 2.1
79 matches
Mail list logo