[MTCGA]: new failures in builds [5669621] needs to be handled

[dpavlov . tasks]

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 If your changes can lead to this failure(s): We're grateful that you were a 
volunteer to make the contribution to this project, but things change and you 
may no longer be able to finalize your contribution.
 Could you respond to this email and indicate if you wish to continue and fix 
test failures or step down and some committer may revert you commit. 

 *New Critical Failure in master MVCC PDS 2 
https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_MvccPds2?branch=%3Cdefault%3E
 Changes may lead to failure were done by 
 - maksim timonin  
https://ci.ignite.apache.org/viewModification.html?modId=908712
 - ibessonov  
https://ci.ignite.apache.org/viewModification.html?modId=908699
 - anton kalashnikov  
https://ci.ignite.apache.org/viewModification.html?modId=908709
 - ilya kazakov  
https://ci.ignite.apache.org/viewModification.html?modId=908694

 - Here's a reminder of what contributors were agreed to do 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute 
 - Should you have any questions please contact dev@ignite.apache.org 

Best Regards,
Apache Ignite TeamCity Bot 
https://github.com/apache/ignite-teamcity-bot
Notification generated at 07:11:17 17-10-2020 

[MTCGA]: new failures in builds [5669601] needs to be handled

[dpavlov . tasks]

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 If your changes can lead to this failure(s): We're grateful that you were a 
volunteer to make the contribution to this project, but things change and you 
may no longer be able to finalize your contribution.
 Could you respond to this email and indicate if you wish to continue and fix 
test failures or step down and some committer may revert you commit. 

 *New Critical Failure in master Platform .NET (Long Running) 
https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_PlatformNetLongRunning?branch=%3Cdefault%3E
 Changes may lead to failure were done by 
 - maksim timonin  
https://ci.ignite.apache.org/viewModification.html?modId=908712
 - ibessonov  
https://ci.ignite.apache.org/viewModification.html?modId=908699
 - anton kalashnikov  
https://ci.ignite.apache.org/viewModification.html?modId=908709
 - ilya kazakov  
https://ci.ignite.apache.org/viewModification.html?modId=908694

 - Here's a reminder of what contributors were agreed to do 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute 
 - Should you have any questions please contact dev@ignite.apache.org 

Best Regards,
Apache Ignite TeamCity Bot 
https://github.com/apache/ignite-teamcity-bot
Notification generated at 06:11:18 17-10-2020 

[MTCGA]: new failures in builds [5669611] needs to be handled

[dpavlov . tasks]

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 If your changes can lead to this failure(s): We're grateful that you were a 
volunteer to make the contribution to this project, but things change and you 
may no longer be able to finalize your contribution.
 Could you respond to this email and indicate if you wish to continue and fix 
test failures or step down and some committer may revert you commit. 

 *New Critical Failure in master MVCC Cache 1 
https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_MvccCache1?branch=%3Cdefault%3E
 Changes may lead to failure were done by 
 - maksim timonin  
https://ci.ignite.apache.org/viewModification.html?modId=908712
 - ibessonov  
https://ci.ignite.apache.org/viewModification.html?modId=908699
 - anton kalashnikov  
https://ci.ignite.apache.org/viewModification.html?modId=908709
 - ilya kazakov  
https://ci.ignite.apache.org/viewModification.html?modId=908694

 - Here's a reminder of what contributors were agreed to do 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute 
 - Should you have any questions please contact dev@ignite.apache.org 

Best Regards,
Apache Ignite TeamCity Bot 
https://github.com/apache/ignite-teamcity-bot
Notification generated at 05:41:32 17-10-2020 

Re: Video overview of eight Ignite-specific talks scheduled for the IMCS Summit 2020

[Saikat Maitra]

Thank you so much Denis for the Video support.

Regards,
Saikat

On Fri, Oct 16, 2020 at 2:03 PM Alexey Zinoviev 
wrote:

> Thanks for the video support, Denis, shared!
>
> пт, 16 окт. 2020 г., 21:45 Denis Magda :
>
>> Igniters,
>>
>> We've got eight (!!!) Ignite-specific sessions scheduled for the
>> In-Memory Computing Summit 2020. I've gone ahead and produced a quick
>> overview of those sessions. Check and weigh up which one to join:
>> https://youtu.be/SCqd4qfBY6Q
>>
>> The sessions are delivered by our community folks: @Saikat Maitra
>> , @Alexey Zinoviev , 
>> @Stanislav
>> Lukyanov , @Andrey Alexandrov
>> , @Glenn Wiebe , @Greg
>> Stachnick , me
>>
>> Support the speakers and share the video within your network.
>>
>> -
>> Denis
>>
>

[MTCGA]: new failures in builds [5668745] needs to be handled

[dpavlov . tasks]

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 If your changes can lead to this failure(s): We're grateful that you were a 
volunteer to make the contribution to this project, but things change and you 
may no longer be able to finalize your contribution.
 Could you respond to this email and indicate if you wish to continue and fix 
test failures or step down and some committer may revert you commit. 

 *New Critical Failure in master Thin client: Node.js 
https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_ThinClientNodeJs?branch=%3Cdefault%3E
 Changes may lead to failure were done by 
 - denis magda  
https://ci.ignite.apache.org/viewModification.html?modId=908686

 - Here's a reminder of what contributors were agreed to do 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute 
 - Should you have any questions please contact dev@ignite.apache.org 

Best Regards,
Apache Ignite TeamCity Bot 
https://github.com/apache/ignite-teamcity-bot
Notification generated at 01:11:17 17-10-2020 

Re: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]

[Denis Magda]

@Alex Plehanov ,

I crossed off the step 6.3.4 (technical docs publication)

of
the release process by adding the new docs to the website's menu. You can
find them by going to the "Resources"->"Documentation" menu item. I will be
unreachable early next week, but it feels like the 2.9 vote will pass;
thus, decided to proceed with this step.

Just in case, here is the documentation contribution and publication
process. Use it if you need to tweak anything:
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document#HowtoDocument-PublishingtotheWebsite


-
Denis


On Mon, Oct 12, 2020 at 12:57 PM Denis Magda  wrote:

> Saikat, Alex,
>
> Based on your inputs here, it sounds like once Ignite 2.9 gets released,
> all the integrations that made their way to the extensions repo [1] will
> get stuck in limbo for some time. What I mean here:
>
>1. While the users will be bumping up their ignite-core,
>ignite-indexing, etc. versions to 2.9, they won't find a 2.9 artifact for
>Kafka, Camel and other extensions
>2. Also, the users won't find 1.0 versions of those extensions that
>also need to be released
>
> So, it's unclear how those users are supposed to upgrade to Ignite 2.9 if
> they use any of the extensions. Is it safe to use a 2.8 version of an
> extension? Or, should we release the 1.0 versions of all the migrated
> extensions together with 2.9?
>
>
> [1] https://github.com/apache/ignite-extensions/tree/master/modules
>
>
> -
> Denis
>
>
> On Tue, Oct 6, 2020 at 7:20 AM Alexey Goncharuk <
> alexey.goncha...@gmail.com> wrote:
>
>> Saikat,
>>
>> The plan sounds good to me! Do you have an idea for the timeline of the
>> module releases? Let me know if I can help in any way.
>>
>> Also, I was looking into the modularization IEP and noticed that OSGI
>> module is discussed to be moved to the extensions, but there is no
>> corresponding ticket. Should I create one? I will be happy to help with
>> moving this module to extensions.
>>
>> вт, 29 сент. 2020 г. в 03:03, Saikat Maitra :
>>
>> > Hi Alexey,
>> >
>> > All the modules as planned in IEP-36
>> >
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36:+Modularization
>> > are migrated to ignite-extensions repository.
>> >
>> > We can definitely plan to release ignite-extensions modules.
>> >
>> > I have a pending PR related to the kafka module and the PR is in review
>> > https://github.com/apache/ignite/pull/8222 but its corresponding PR has
>> > been merged in ignite-extensions repository.
>> >
>> > My thoughts / action items for the migration efforts and releases were
>> > as follows:
>> >
>> > 1. Merge the changes for https://github.com/apache/ignite/pull/8222
>> > 2. Fix the upload nightly packages task for ignite-core to help fix the
>> > travis build failure in ignite-extensions repository.
>> > 3. Review and update the README.md files for the ignite-extensions
>> modules
>> > as required.
>> > 4. Update the docs for ignite-extensions modules in ignite-website.
>> > 5. Release each module separately and share updates.
>> >
>> > Please let me know if you have feedback.
>> >
>> > Regards,
>> > Saikat
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Mon, Sep 28, 2020 at 7:36 AM Petr Ivanov 
>> wrote:
>> >
>> >> It seems that gitbox.apache.org is not available on CI/CD.
>> >>
>> >> I will try to update VCS to GitHub.
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> > On 28 Sep 2020, at 14:07, Alex Plehanov 
>> >> wrote:
>> >> >
>> >> > Guys,
>> >> >
>> >> > I've tried to build release candidate using TC [1]. But:
>> >> > 1. I can't choose a branch in this TC task (there is no such field).
>> >> > 2. On the "default" branch I got an error: Failed to collect changes,
>> >> > error: List remote refs failed:
>> >> > org.apache.http.conn.ConnectTimeoutException: Connect to
>> >> > gitbox.apache.org:443 [gitbox.apache.org/52.202.80.70] failed:
>> connect
>> >> > timed out, VCS root: "GitBox [asf/ignite]" {instance id=300, parent
>> >> > internal id=74, parent id=GitBoxAsfIgnite, description: "
>> >> > https://gitbox.apache.org/repos/asf/ignite.git#refs/heads/master"}
>> >> >
>> >> > Is there some problem with this TC task? What am I doing wrong?
>> >> >
>> >> > [1] :
>> >> >
>> >>
>> https://ci.ignite.apache.org/viewType.html?buildTypeId=Releases_ApacheIgniteMain_ReleaseBuild
>> >> >
>> >> > пн, 28 сент. 2020 г. в 13:15, Alexey Goncharuk <
>> >> alexey.goncha...@gmail.com>:
>> >> >
>> >> >> Saikat, Nikolay,
>> >> >>
>> >> >> We have migrated a bunch of modules to ignite-extensions recently.
>> >> Given
>> >> >> that these modules will not be available in Ignite 2.9 anymore (will
>> >> >> they?), should we also plan to release the extensions?
>> >> >>
>> >> >> ср, 23 сент. 2020 г. в 21:49, Alex Plehanov <
>> plehanov.a...@gmail.com>:
>> >> >>
>> >> >>> Igniters,
>> >> >>>
>> >> >>> I've prepared release notes for the 2.9 release 

[jira] [Created] (IGNITE-13589) Ignite Docs: the docs root must open the latest version

[Denis A. Magda (Jira)]

Denis A. Magda created IGNITE-13589:
---

 Summary: Ignite Docs: the docs root must open the latest version
 Key: IGNITE-13589
 URL: https://issues.apache.org/jira/browse/IGNITE-13589
 Project: Ignite
  Issue Type: Task
  Components: documentation
Reporter: Denis A. Magda
 Fix For: 2.10


If you open the default docs URL (https://ignite.apache.org/docs/), you'll see 
a structure of a directory on the server.

That default URL needs to redirect to https://ignite.apache.org/docs/latest/



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Video overview of eight Ignite-specific talks scheduled for the IMCS Summit 2020

[Alexey Zinoviev]

Thanks for the video support, Denis, shared!

пт, 16 окт. 2020 г., 21:45 Denis Magda :

> Igniters,
>
> We've got eight (!!!) Ignite-specific sessions scheduled for the In-Memory
> Computing Summit 2020. I've gone ahead and produced a quick overview of
> those sessions. Check and weigh up which one to join:
> https://youtu.be/SCqd4qfBY6Q
>
> The sessions are delivered by our community folks: @Saikat Maitra
> , @Alexey Zinoviev , 
> @Stanislav
> Lukyanov , @Andrey Alexandrov
> , @Glenn Wiebe , @Greg
> Stachnick , me
>
> Support the speakers and share the video within your network.
>
> -
> Denis
>

Video overview of eight Ignite-specific talks scheduled for the IMCS Summit 2020

[Denis Magda]

Igniters,

We've got eight (!!!) Ignite-specific sessions scheduled for the In-Memory
Computing Summit 2020. I've gone ahead and produced a quick overview of
those sessions. Check and weigh up which one to join:
https://youtu.be/SCqd4qfBY6Q

The sessions are delivered by our community folks: @Saikat Maitra
, @Alexey Zinoviev ,
@Stanislav
Lukyanov , @Andrey Alexandrov
, @Glenn Wiebe , @Greg
Stachnick , me

Support the speakers and share the video within your network.

-
Denis

[DISCUSS] Use Netty for Java thin client

[Pavel Tupitsyn]

Igniters,

I'm working on IEP-51 [1] to make Java thin client truly async
and make sure user threads are never blocked
(right now socket writes are performed from user threads).

I've investigated potential approaches and came to the conclusion
that Netty [2] is our best bet.
- Nice Future-based async API => will greatly reduce our code complexity
  and remove manual thread management
- Potentially reduced resource usage - share EventLoopGroop across all
connections within one IgniteClient
- SSL is easy to use
- Proven performance and reliability

Other approaches, like AsynchronousSocketChannel or selectors, seem to be
too complicated,
especially when SSL comes into play.
We should focus on Ignite-specific work instead of spending time on
reinventing async IO.

The obvious downside is an extra dependency in the core module.
However, I heard some discussions about using Netty for GridNioServer in
future.

Let me know your thoughts.

Pavel

[1]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-51%3A+Java+Thin+Client+Async+API
[2] https://netty.io

[jira] [Created] (IGNITE-13588) Returns the full name of the specified type without any assembly version info. The reason why this method is needed is that a generic type's FullName contains the full

[Danut Radoaica (Jira)]

Danut Radoaica created IGNITE-13588:
---

 Summary: Returns the full name of the specified type without any 
assembly version info. The reason why this method is needed is that a generic 
type's FullName contains the full AssemblyQualifiedName of its item type. 
 Key: IGNITE-13588
 URL: https://issues.apache.org/jira/browse/IGNITE-13588
 Project: Ignite
  Issue Type: Bug
  Components: platforms
Affects Versions: 2.8.1
 Environment: Apache Ignite: v2.8.1

JDK: v1.8

.NET Core: v3.1
Reporter: Danut Radoaica
 Attachments: Untitled.png

Returns the full name of the specified type without any assembly version info. 
The reason why this method is needed is that a generic type's FullName contains 
the full AssemblyQualifiedName of its item type.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSSION] IEP-59: CDC - Capture Data Change

[Pavel Kovalenko]

Alexey,

>> If a CDC agent is restarted, it will have to start from scratch
>> If a CDC reader does not keep up with the WAL write rate (e.g. there
   is a short-term write burst and WAL archive is small), the Ignite node
will
   delete WAL segments while the consumer is still reading it.

I think these cases can be resolved with the following approach:
PostgreSQL can be configured to execute a shell command after WAL segment
is archived. The same thing we can do for Ignite as well.
A command can create a hardlink for such WAL segment to a specified
directory to not loose it after deletion by Ignite and notify a CDC (or
another kind of process) about this segment.
That will be a filesystem queue and CDC after restart may proceed only
segments located at this directory, so it's no need to start from scratch.
When WAL segment is processed by CDC a hardlink from queue directory is
deleted.



пт, 16 окт. 2020 г. в 13:42, Alexey Goncharuk :

> Hello Nikolay,
>
> Thanks for the suggestion, it definitely may be a good feature, however, I
> do not see any significant value that it currently adds to the already
> existing WAL Iterator. I think the following issues should be addressed,
> otherwise, no regular user will be able to use the CDC reliably:
>
>- The interface exposes WALRecord which is a private API
>- There is no way to start capturing changes from a certain point (a
>watermark for already processed data). Users can configure a large size
> for
>WAL archive to sustain long node downtime for historical rebalance. If a
>CDC agent is restarted, it will have to start from scratch. I see that
> it
>is present in the IEP as a design choice, but I think this is a major
>usability issue
>- If a CDC reader does not keep up with the WAL write rate (e.g. there
>is a short-term write burst and WAL archive is small), the Ignite node
> will
>delete WAL segments while the consumer is still reading it. Since the
>consumer is running out-of-process, we need to specify some sort of
>synchronization protocol between the node and the consumer
>- If Ignite node crashes, gets restarted and initiates full rebalance,
>the consumer will lose some updates
>- Usually, it makes sense for the CDC consumer to read updates only on
>primary nodes (otherwise, multiple agents will be doing duplicate
> work). In
>the current design, the consumer will not be able to differentiate
>primary/backup updates. Moreover, even if we wrote such flags to WAL,
> the
>consumer would need to process backup records anyway because it is
> unknown
>whether the primary consumer is alive. In other words, how would an end
>user organize the CDC failover minimizing the duplicate work?
>
>
> ср, 14 окт. 2020 г. в 14:21, Nikolay Izhikov :
>
> > Hello, Igniters.
> >
> > I want to start a discussion of the new feature [1]
> >
> > CDC - capture data change. The feature allows the consumer to receive
> > online notifications about data record changes.
> >
> > It can be used in the following scenarios:
> > * Export data into some warehouse, full-text search, or
> > distributed log system.
> > * Online statistics and analytics.
> > * Wait and respond to some specific events or data changes.
> >
> > Propose to implement new IgniteCDC application as follows:
> > * Run on the server node host.
> > * Watches for the appearance of the WAL archive segments.
> > * Iterates it using existing WALIterator and notifies consumer of
> > each record from the segment.
> >
> > IgniteCDC features:
> > * Independence from the server node process (JVM) - issues and
> > failures of the consumer will not lead to server node instability.
> > * Notification guarantees and failover - i.e. CDC track and save
> > the pointer to the last consumed record. Continue notification from this
> > pointer in case of restart.
> > * Resilience for the consumer - it's not an issue when a consumer
> > temporarily consumes slower than data appear.
> >
> > WDYT?
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-59+CDC+-+Capture+Data+Change
>

Re: [DISCUSSION] IEP-59: CDC - Capture Data Change

[Alexey Goncharuk]

Hello Nikolay,

Thanks for the suggestion, it definitely may be a good feature, however, I
do not see any significant value that it currently adds to the already
existing WAL Iterator. I think the following issues should be addressed,
otherwise, no regular user will be able to use the CDC reliably:

   - The interface exposes WALRecord which is a private API
   - There is no way to start capturing changes from a certain point (a
   watermark for already processed data). Users can configure a large size for
   WAL archive to sustain long node downtime for historical rebalance. If a
   CDC agent is restarted, it will have to start from scratch. I see that it
   is present in the IEP as a design choice, but I think this is a major
   usability issue
   - If a CDC reader does not keep up with the WAL write rate (e.g. there
   is a short-term write burst and WAL archive is small), the Ignite node will
   delete WAL segments while the consumer is still reading it. Since the
   consumer is running out-of-process, we need to specify some sort of
   synchronization protocol between the node and the consumer
   - If Ignite node crashes, gets restarted and initiates full rebalance,
   the consumer will lose some updates
   - Usually, it makes sense for the CDC consumer to read updates only on
   primary nodes (otherwise, multiple agents will be doing duplicate work). In
   the current design, the consumer will not be able to differentiate
   primary/backup updates. Moreover, even if we wrote such flags to WAL, the
   consumer would need to process backup records anyway because it is unknown
   whether the primary consumer is alive. In other words, how would an end
   user organize the CDC failover minimizing the duplicate work?


ср, 14 окт. 2020 г. в 14:21, Nikolay Izhikov :

> Hello, Igniters.
>
> I want to start a discussion of the new feature [1]
>
> CDC - capture data change. The feature allows the consumer to receive
> online notifications about data record changes.
>
> It can be used in the following scenarios:
> * Export data into some warehouse, full-text search, or
> distributed log system.
> * Online statistics and analytics.
> * Wait and respond to some specific events or data changes.
>
> Propose to implement new IgniteCDC application as follows:
> * Run on the server node host.
> * Watches for the appearance of the WAL archive segments.
> * Iterates it using existing WALIterator and notifies consumer of
> each record from the segment.
>
> IgniteCDC features:
> * Independence from the server node process (JVM) - issues and
> failures of the consumer will not lead to server node instability.
> * Notification guarantees and failover - i.e. CDC track and save
> the pointer to the last consumed record. Continue notification from this
> pointer in case of restart.
> * Resilience for the consumer - it's not an issue when a consumer
> temporarily consumes slower than data appear.
>
> WDYT?
>
> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-59+CDC+-+Capture+Data+Change

[jira] [Created] (IGNITE-13587) Calcite improvements. Append tests for unstable topology checking.

[Stanilovsky Evgeny (Jira)]

Stanilovsky Evgeny created IGNITE-13587:
---

 Summary: Calcite improvements. Append tests for unstable topology 
checking.
 Key: IGNITE-13587
 URL: https://issues.apache.org/jira/browse/IGNITE-13587
 Project: Ignite
  Issue Type: Test
  Components: sql
Affects Versions: 2.8.1
Reporter: Stanilovsky Evgeny
Assignee: Stanilovsky Evgeny


After [1] was found in h2 realization we probably need for such kind of tests 
with calcite.
[1] https://issues.apache.org/jira/browse/IGNITE-13572 (Duplicates in select 
query during partition eviction.)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re[2]: IEP-58: Statistics

[Zhenya Stanilovsky]

Andrey, thanks for firing this ! 
Sasha it`s unclear for me « These part consists of two processes: statistics 
collection process itself and acquiring statistics by the client. »:
*  I agree that in both cases local statistics are useless.
May be we need more informative use cases for such statistics usage ? Can 
someone append additional columns (possible not presented in index) statistics? 
*  Client — can you unfold this term ?  If this means — ignite client node ? 
Does sql best plan is chosen in request starter node ? If so — what about this 
client with limited cpu here? 
*  « If there are no statistics in all of them - client will choose random   » 
— not random but affinity concerted isn`t it ? 
*  « After getting statistics client will cache it and server node it to renew 
statistics from same node. ​​​»  I don`t understand this approach, can you 
clarify it plz ?
*  Whats the storage mechanism for client node statistics?
*  Can we use thin client without discs in such cases?
thanks !
  
>:
> 
>Follow up
>
>Igniters,
>
>is there any comment to this IEP?
>
>JFYI, IEP is renamed and placed here [1]
>
>[1]  
>https://cwiki.apache.org/confluence/display/IGNITE/IEP-58%3A+Statistics+for+SQL+query+optimization
>
>On Thu, Sep 24, 2020 at 2:30 PM Sasha Belyak < rtsfo...@gmail.com > wrote:
>>
>> Igniters,
>> I'e prepared an IEP [1], please review and let me know what you think.
>>
>> In particular, I'd like to discuss the new subsystem to collect statistics
>> to optimize sql queries execution.
>> [1]  https://cwiki.apache.org/confluence/display/IGNITE/IEP-58+Statistics