[jira] [Created] (IGNITE-9956) Web Agent should handle EOFException in the process of node stopping

[Roman Guseinov (JIRA)]

Roman Guseinov created IGNITE-9956:
--

 Summary: Web Agent should handle EOFException in the process of 
node stopping
 Key: IGNITE-9956
 URL: https://issues.apache.org/jira/browse/IGNITE-9956
 Project: Ignite
  Issue Type: Bug
  Components: UI
Reporter: Roman Guseinov


If web agent is connected to a node which starts stopping we will see the 
following message in the agent's logs:

{code:java}
[2018-10-03 12:19:32,672][ERROR][pool-3-thread-48][AbstractListener] Failed to 
execute REST command with parameters: {p1=6840C2E6-EE22-4F96-8BC3-35DFD60497CF, 
p2=org.gridgain.grid.internal.visor.node.VisorGridGainNodeConfigurationCollectorTask,
 p3=java.lang.Void, name=o.a.i.i.v.compute.VisorGatewayTask, cmd=exe}
java.io.IOException: unexpected end of stream on 
Connection{datafabric-dev-22.test.com:8080, proxy=DIRECT 
hostAddress=datafabric-dev-22.test.com/10.54.1.21:8080 cipherSuite=none 
protocol=http/1.1}
at 
okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:205)
at 
okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75)
at 
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at 
okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
at 
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at 
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at 
okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
at 
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at 
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at 
okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at 
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at 
okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
at 
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at 
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
at okhttp3.RealCall.execute(RealCall.java:69)
at 
org.apache.ignite.console.agent.rest.RestExecutor.sendRequest(RestExecutor.java:169)
at 
org.apache.ignite.console.agent.rest.RestExecutor.sendRequest(RestExecutor.java:184)
at 
org.apache.ignite.console.agent.handlers.RestListener.execute(RestListener.java:85)
at 
org.apache.ignite.console.agent.handlers.AbstractListener.lambda$call$0(AbstractListener.java:76)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException: \n not found: limit=0 content=…
at 
okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:226)
at 
okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:210)
at 
okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
... 24 more
{code}

It looks like node returns an empty response. Web agent should handle that, 
print a more understandable message and start trying to connect to other nodes 
immediately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

How to begin to contribute

[Michael Fong]

Hi, all,

I happened to have a fix for IGNITE-7153 and created pull request for it. (
https://github.com/apache/ignite/pull/5044)

I tried to follow the How to Contribute Workflow

but
kind of got stuck on TeamCity CI, where all the test suite build failed at
unrelated library (ignite-tool). Is there any other easier or more
intuitive CI or documentation to guide through the testing process?

Furthermore, I wonder what would the code review process be, still via
github comment review?

Regards,

Michael

[GitHub] ignite pull request #5044: IGNITE-7153: Fix parser to handle incomplete Redi...

[mcfongtw]

GitHub user mcfongtw opened a pull request:

https://github.com/apache/ignite/pull/5044

IGNITE-7153: Fix parser to handle incomplete Redis packet

Add validation method to check packet completeness

Add logic to store incomplete packet temporarily and assemble them until
the final CRLF is seen.

Implement test cases for data = 8k, 10k and 16k

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mcfongtw/ignite ignite-7153

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/5044.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5044






---

[jira] [Created] (IGNITE-9955) Fail of cache creation on cluster activation

[Denis Garus (JIRA)]

Denis Garus created IGNITE-9955:
---

 Summary: Fail of cache creation on cluster activation
 Key: IGNITE-9955
 URL: https://issues.apache.org/jira/browse/IGNITE-9955
 Project: Ignite
  Issue Type: Bug
Reporter: Denis Garus
 Attachments: CacheStartFailOnClusterActivate.java

If an error has occurred during cluster activation (processing of 
ChangeGlobalStateMessage) then no caches will be created, including system 
cache.

The reproducer is attached.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: New TC builds adding process

[Dmitriy Pavlov]

Hi Ivan,

I believe every time any suite is added or significantly reconfigured it
should be mentioned in the dev list.

So we don't need to approve any change by discussion, but keeping community
member posted seems to be really helpful here.

Currently, I don't know why and which suites were added.

Sincerely,
Dmitriy Pavlov

сб, 20 окт. 2018 г. в 9:25, Павлухин Иван :

> Hi Igniters,
>
> On Friday I felt a real inconvenience (pain in ass) with running builds for
> a ticked I worked on [1]. There were two the most frustrating moments:
> 1. Estimated build completion time was 11 hours after the start.
> 2. There are newly added suites which are failing constantly in recent
> runs.
>
> I compared builds count in RunAll chain for mentioned ticked. On Oct 6
> there were 105 builds, and on Oct 19 where were 113. Then I compared
> differences and found suspicious items.
>
> First of all there were 2 Windows suites PlatformNetCoverage [2] and
> PlatformCWindowsX86 [3]. Former seems to be something added recently and
> latter as I know was decided to be excluded from RunAll some time ago but
> strangely reappeared. I suspect that these 2 suites could increase build
> run time because they are quite lengthy and require Windows slaves which
> are in limited amount.
>
> Also there were 2 rather problematic builds InspectionsAop [4] and
> InspectionsCore [5] which seems to fail constantly. Presence of such builds
> brings a noise into analyzing build results. TC Bot treats them as possible
> blockers.
>
> So, here are my suggestions:
> 1. A contributor should add new build to RunAll after estimating build
> running time impact carefully. If impact is noticeable such builds should
> be announced and discussed on dev list. It sounds good idea to test drive
> new heavy build running it on a scheduled basis first before adding to
> RunAll.
> 2. We should not have constantly failing builds like mentioned Inspections.
>
> What are your thoughts?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-5935
> [2]
>
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_PlatformNetCoverage
> [3]
>
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_PlatformCWindowsX86
> [4]
>
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_InspectionsAop
> [5]
>
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_InspectionsCore
>
> --
> Best regards,
> Ivan Pavlukhin
>

Re: Abbreviation code-style requirement.

[Dmitriy Pavlov]

+1 for proposal 3.

1. I'm not sure we need to revisit all abbreviations as a lot of people get
used to it.
2. I'm not sure multiword is always need to be fully named, sometimes it
may be ok to abbreviate.
3. But I agree with abbreviations should not be mandatory.

Abbreviated and short names like i,j,cp and etc. are good for simple
methods and code blocks; for a fast demonstration of some idea, but for
complex enterprise level software it can hide meaning instead of clearly
showing it.

As a next step, I would like to propose to contribute an option to disable
abbreviation requirements for some cases in ignite-abbrev-plugin.

сб, 20 окт. 2018 г. в 10:47, Zhenya :

> +1 for all proposals.
>

Re: Apache Ignite 2.7. Last Mile

[Nikolay Izhikov]

Hell, Denis.

I just filter all 2.7 tickets without documentation.


https://issues.apache.org/jira/browse/IGNITE-9932?jql=(project%20%3D%20%27Ignite%27%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20(%272.7%27)%20AND%20status%20NOT%20IN%20(Resolved%2C%20Closed)%20and%20(component%20is%20null%20or%20component%20NOT%20IN%20(documentation)))%20ORDER%20BY%20%20priority%20%20%20%20%20%20%20%20%20%20%20%20%20%20

I've added this view to the release page.
See sesction "Unresolved tickets(without documentation)".

В Сб, 20/10/2018 в 16:04 -0700, Denis Magda пишет:
> Nikolay,
> 
> Where do you track those 8 blockers? Can't find them on this wiki page:
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.7
> 
> --
> Denis
> 
> On Sat, Oct 20, 2018 at 11:31 AM Nikolay Izhikov 
> wrote:
> 
> > Hello, Denis.
> > 
> > As a first time release manager I'm trying to rely on Ignite veterans
> > opinion.
> > Guys told me that we must fix all blockers and only after it make the
> > release.
> > 
> > Let's fix them all.
> > What do you think?
> > 
> > > When are we sending a release candidate for vote?
> > 
> > When all blocker bugs will be fixed.
> > Currently, we have *8*.
> > 
> > В Пт, 19/10/2018 в 13:50 -0700, Denis Magda пишет:
> > > Guys, as a side observer of the current release, this all looks like a
> > > never ending story :)
> > > 
> > > When are we sending a release candidate for vote?
> > > 
> > > --
> > > Denis
> > > 
> > > On Fri, Oct 19, 2018 at 4:39 AM Nikolay Izhikov 
> > 
> > wrote:
> > > 
> > > > Hello, Igniters.
> > > > 
> > > > We have 6 tickets for 2.7
> > > > 
> > > > Roman Kondakov - IGNITE-9892, IGNITE-9663, IGNITE-9928
> > > > Igor Seliverstov - IGNITE-9911
> > > > Ivan Pavlukhin - IGNITE-5935, IGNITE-9944
> > > > 
> > > > В Чт, 18/10/2018 в 15:40 +0300, Andrey Kuznetsov пишет:
> > > > > I have got one more potential 2.7 blocker [1] with straightforward
> > 
> > fix. I
> > > > > beleive it will not break any production use case, but it leads to
> > 
> > test
> > > > > suite hang, thus affecting other urgent issues.
> > > > > 
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-9932
> > > > > 
> > > > > чт, 18 окт. 2018 г. в 14:59, Ivan Daschinsky :
> > > > > 
> > > > > > Hi! Is it possible to merge IGNITE-9854? Fix is pretty simple, but
> > > > 
> > > > quite
> > > > > > important.
> > > > > > 
> > > > > > ср, 17 окт. 2018 г. в 17:49, Andrey Gura :
> > > > > > 
> > > > > > > JFYI
> > > > > > > 
> > > > > > > IGNITE-9737 and IGNITE-9710 are merged to release branch.
> > > > > > > On Wed, Oct 17, 2018 at 5:41 PM Pavel Tupitsyn <
> > 
> > ptupit...@apache.org
> > > > > > > wrote:
> > > > > > > > 
> > > > > > > > Thank you. Fix has been merged to master and cherry-picked to
> > > > > > 
> > > > > > ignite-2.7.
> > > > > > > > 
> > > > > > > > On Wed, Oct 17, 2018 at 1:26 PM Nikolay Izhikov <
> > > > 
> > > > nizhi...@apache.org>
> > > > > > > 
> > > > > > > wrote:
> > > > > > > > 
> > > > > > > > > Pavel.
> > > > > > > > > 
> > > > > > > > > Ok, I agree to include this ticket into 2.7
> > > > > > > > > Let's do it.
> > > > > > > > > 
> > > > > > > > > В Ср, 17/10/2018 в 13:20 +0300, Pavel Tupitsyn пишет:
> > > > > > > > > > Nikolay,
> > > > > > > > > > 
> > > > > > > > > > It completely breaks a major feature under certain
> > 
> > conditions.
> > > > 
> > > > I
> > > > > > > 
> > > > > > > would
> > > > > > > > > > consider it a blocker.
> > > > > > > > > > 
> > > > > > > > > > On Wed, Oct 17, 2018 at 1:00 PM Nikolay Izhikov <
> > > > > > 
> > > > > > nizhi...@apache.org
> > > > > > > > 
> > > > > > > > > wrote:
> > > > > > > > > > 
> > > > > > > > > > > Hello, Pavel.
> > > > > > > > > > > 
> > > > > > > > > > > Is it a blocker?
> > > > > > > > > > > 
> > > > > > > > > > > В Ср, 17/10/2018 в 12:58 +0300, Pavel Tupitsyn пишет:
> > > > > > > > > > > > Hi Igniters,
> > > > > > > > > > > > 
> > > > > > > > > > > > I'd like to include IGNITE-9877 in 2.7, can we do that?
> > > > > > > > > > > > The fix is ready, I'm waiting for TC run.
> > > > > > > > > > > > 
> > > > > > > > > > > > Pavel
> > > > > > > > > > > > 
> > > > > > > > > > > > On Wed, Oct 17, 2018 at 11:45 AM Павлухин Иван <
> > > > > > > 
> > > > > > > vololo...@gmail.com>
> > > > > > > > > > > 
> > > > > > > > > > > wrote:
> > > > > > > > > > > > 
> > > > > > > > > > > > > Hi NIkolay,
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Thank you for keeping everybody focused! Regarding
> > 
> > to my
> > > > > > 
> > > > > > ticket
> > > > > > > > > > > > > IGNITE-5935.
> > > > > > > > > > > > > It is in final stage now. Tests look good. I believe
> > > > 
> > > > that it
> > > > > > > 
> > > > > > > will
> > > > > > > > > be
> > > > > > > > > > > 
> > > > > > > > > > > merged
> > > > > > > > > > > > > in couple of days (at most).
> > > > > > > > > > > > > 
> > > > > > > > > > > > > ср, 17 окт. 2018 г. в 11:39, Nikolay Izhikov <
> > > > > > > 
> > > > > > > nizhi...@apache.org
> > > > > > > > > > :
> > >

Re: [DISCUSSION] Spark Data Frame through Thin Client

[Nikolay Izhikov]

Valentin.

Seems, You made several suggestions, which is not always true, from my point of 
view:

1. "We have access to Spark cluster installation to perform deployment steps" - 
this is not true in cloud or enterprise environment.

2. "Spark cluster is used only for Ignite integration".
From what I know computational resources for big Spark cluster is divided by 
many business divisions.
And it is not convenient to perform some deployment steps on this cluster.

3. "When Ignite + Spark are used in real production it's OK to have reasonable 
deployment overhead"
What about developer who want to play with this integration?
And want to do it quickly to see how it works in real life examples.
Can we do his life much easier?

> First of all, they will exist with thin client either.

Spark have an ability to deploy jars on worker and add it to application tasks 
classpath.
For 2.6 we must deploy 11 additional jars to start using Ignite.
Please, see my example on the bottom of documentation page [1]

Does cache-api-1.0.0.jar and h2-1.4.195.jar seems like obvious dependencies for 
Ignite integration for you?
And for our users? :)

Actually, list of dependencies will be changed in 2.7 - new version of jcache, 
new version of h2
So user should change it in code or perform additional deployment steps.

It overkill for me.

On the other hand - thin client requires only 1 jar.
Moreover, thin client protocol have the backward compatibility.
So thin client will perform correctly when Ignite cluster will be updated from 
2.6 to 2.7.
So, with Spark integration via thin client we will be able to update Ignite 
cluster and Spark integration separately.
For now, we should do it in one big step.

What do you think?

[1] https://apacheignite-fs.readme.io/docs/installation-deployment

В Сб, 20/10/2018 в 18:33 -0700, Valentin Kulichenko пишет:
> Guys,
> 
> From my experience, Ignite and Spark clusters typically run in the same
> environment, which makes client node a more preferable option. Mainly,
> because of performance. BTW, I doubt partition-awareness on thin client
> will help either, because in dataframes we only run SQL queries and I
> believe thin client will execute them through a proxy anyway. But correct
> me if I’m wrong.
> 
> Either way, it sounds like we just have usability issues with Ignite/Spark
> integration. Why don’t we concentrate on fixing them then? For example, #3
> can be fixed by loading XML content on master and then distributing it to
> workers, instead of loading on every worker independently. Then there are
> certain procedures like deploying JARs, etc. First of all, they will exist
> with thin client either. Second of all, I’m sure there are ways to simplify
> this procedures and make integration easier. My opinion is that working on
> such improvements is going to add more value than another implementation
> based on thin client.
> 
> -Val
> 
> On Sat, Oct 20, 2018 at 4:03 PM Denis Magda  wrote:
> 
> > Hello Nikolay,
> > 
> > Your proposal sounds reasonable. However, I would suggest us to wait while
> > partition-awareness is supported for Java thin client first. With that
> > feature, the client can connect to any node directly while presently all
> > the communication goes through a proxy (a node the client is connected to).
> > All of that is bad for performance.
> > 
> > 
> > Vladimir, how hard would it be to support the partition-awareness for Java
> > client? Probably, Nikolay can take over.
> > 
> > --
> > Denis
> > 
> > 
> > On Sat, Oct 20, 2018 at 2:09 PM Nikolay Izhikov 
> > wrote:
> > 
> > > Hello, Igniters.
> > > 
> > > Currently, Spark Data Frame integration implemented via client node
> > > connection.
> > > Whenever we need to retrieve some data into Spark worker(or master) from
> > > Ignite we start a client node.
> > > 
> > > It has several major disadvantages:
> > > 
> > > 1. We should copy whole Ignite distribution on to each Spark
> > > worker [1]
> > > 2. We should copy whole Ignite distribution on to Spark master to
> > > get catalogue works.
> > > 3. We should have the same absolute path to Ignite configuration
> > > file on every worker and provide it during data frame construction [2]
> > > 4. We should additionally configure Spark workerks classpath to
> > > include Ignite libraries.
> > > 
> > > For now, almost all operation we need to do in Spark Data Frame
> > > integration is supported by Java Thin Client.
> > > * obtain the list of caches.
> > > * get cache configuration.
> > > * execute SQL query.
> > > * stream data to the table - don't support by the thin client for
> > > now, but can be implemented using simple SQL INSERT statements.
> > > 
> > > Advantages of usage Java Thin Client in Spark integration(they all known
> > > from Java Thin Client advantages):
> > > 1. Easy to configure: only IP addresses of server nodes are
> > > required.
> > > 2. Easy to deploy: only 1 additional j