[GitHub] ignite pull request #1290: Ignite 4312

2016-11-27 Thread alexpaschenko 
GitHub user alexpaschenko opened a pull request:

https://github.com/apache/ignite/pull/1290

Ignite 4312



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-4312

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1290.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1290


commit 6e35c0752b1337846bddf427032e3878c9d0c3ab
Author: Alexander Paschenko 
Date:   2016-11-27T17:52:50Z

Renames and duplicate code removal (trivial)

commit d8bb60b3a0f55b63e6406af30291bc713ba804d5
Author: Alexander Paschenko 
Date:   2016-11-27T19:31:14Z

Test fix

commit 44d011b1a421d6d8f3d39e87f4f31ee7b06a3106
Author: Alexander Paschenko 
Date:   2016-11-28T07:54:26Z

IGNITE-4312 Explicitly copy field values to builders in INSERT/MERGE and 
UPDATE.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-4315) visorcmd: 'cache -swap' shows result for cache that has no swap configuration

2016-11-27 Thread Pavel Konstantinov (JIRA) 
Pavel Konstantinov created IGNITE-4315:
--

 Summary: visorcmd: 'cache -swap' shows result for cache that has 
no swap configuration
 Key: IGNITE-4315
 URL: https://issues.apache.org/jira/browse/IGNITE-4315
 Project: Ignite
  Issue Type: Bug
  Components: visor
Affects Versions: 1.8
Reporter: Pavel Konstantinov


I have a grid with several caches including replicated.
This cache has no swap options in it's configuration, but when I invoke cache 
-swap  I see some entries was swapped.
Here is result output 
{code}
Swapped entries in cache: tx_replicated
++
|  Node ID8(@)  | Entries Swapped | Cache Size Before | Cache Size After |
++
| E9B8330E(@n1) | 12097   | 25092 | 12995|
| AF956783(@n0) | 12992   | 25090 | 12098|
++
{code}

Here is a cache configuration
{code}
visor> cache -config -c=@c29 -a -id8=@n0
ID8=AF956783, time of the snapshot: 11/28/16, 11:20:54
+=+
|   Name(@)   |Mode| Nodes |Entries (Heap / Off-heap)|  
Hits  | Misses | Reads  | Writes  |
+=+
| tx_replicated(@c29) | REPLICATED | 1 | min: 21149 (21149 / 0)  | 
min: 363516| min: 400407| min: 763923| min: 1100927|
| ||   | avg: 21149.00 (21149.00 / 0.00) | 
avg: 363516.00 | avg: 400407.00 | avg: 763923.00 | avg: 1100927.00 |
| ||   | max: 21149 (21149 / 0)  | 
max: 363516| max: 400407| max: 763923| max: 1100927|
+-+

Cache 'tx_replicated(@c29)':
++
| Name(@) | tx_replicated(@c29)  |
| Nodes   | 1|
| Total size Min/Avg/Max  | 21149 / 21149.00 / 21149 |
|   Heap size Min/Avg/Max | 21149 / 21149.00 / 21149 |
|   Off-heap size Min/Avg/Max | 0 / 0.00 / 0 |
++

Nodes for: tx_replicated(@c29)
++
| Node ID8(@), IP  | CPUs | Heap Used | CPU Load |   Up Time|   
  Size | Hi/Mi/Rd/Wr |
++
| AF956783(@n0), 127.0.0.1 | 8| 68.92 %   | 3.50 %   | 01:46:56:004 | 
Total: 21149 | Hi: 363516  |
|  |  |   |  |  |   
Heap: 21149| Mi: 400407  |
|  |  |   |  |  |   
Off-Heap: 0| Rd: 763923  |
|  |  |   |  |  |   
Off-Heap Memory: 0 | Wr: 1100927 |
++
'Hi' - Number of cache hits.
'Mi' - Number of cache misses.
'Rd' - number of cache reads.
'Wr' - Number of cache writes.

Aggregated queries metrics:
  Minimum execution time: 00:00:00:000
  Maximum execution time: 00:00:00:000
  Average execution time: 00:00:00:000
  Total number of executions: 0
  Total number of failures:   0

Cache configuration:
+==+
|   Name|Value  
   |
+==+
| Mode  | REPLICATED
   |
| Atomicity Mode| TRANSACTIONAL 
   |
| Atomic Write Ordering Mode|  
   |
| Statistic Enabled | on
   |
| Management Enabled| off   
   |
| Time To Live Eager Flag   | true

Re: ignite-spark module in Hadoop Accelerator

2016-11-27 Thread Dmitriy Setrakyan 
Separate edition for the Hadoop Accelerator was primarily driven by the
default libraries. Hadoop Accelerator requires many more libraries as well
as configuration settings compared to the standard fabric download.

Now, as far as spark integration is concerned, I am not sure which edition
it belongs in, Hadoop Accelerator or standard fabric.

D.

On Sat, Nov 26, 2016 at 7:39 PM, Denis Magda  wrote:

> *Dmitriy*,
>
> I do believe that you should know why the community decided to a separate
> edition for the Hadoop Accelerator. What was the reason for that?
> Presently, as I see, it brings more confusion and difficulties rather then
> benefit.
>
> —
> Denis
>
> On Nov 26, 2016, at 2:14 PM, Konstantin Boudnik  wrote:
>
> In fact I am very much agree with you. Right now, running the "accelerator"
> component in Bigtop disto gives one a pretty much complete fabric anyway.
> But
> in order to make just an accelerator component we perform quite a bit of
> woodoo magic during the packaging stage of the Bigtop build, shuffling jars
> from here and there. And that's quite crazy, honestly ;)
>
> Cos
>
> On Mon, Nov 21, 2016 at 03:33PM, Valentin Kulichenko wrote:
>
> I tend to agree with Denis. I see only these differences between Hadoop
> Accelerator and Fabric builds (correct me if I miss something):
>
>   - Limited set of available modules and no optional modules in Hadoop
>   Accelerator.
>   - No ignite-hadoop module in Fabric.
>   - Additional scripts, configs and instructions included in Hadoop
>   Accelerator.
>
> And the list of included modules frankly looks very weird. Here are only
> some of the issues I noticed:
>
>   - ignite-indexing and ignite-spark are mandatory. Even if we need them
>   for Hadoop Acceleration (which I doubt), are they really required or can
> be
>   optional?
>   - We force to use ignite-log4j module without providing other logger
>   options (e.g., SLF).
>   - We don't include ignite-aws module. How to use Hadoop Accelerator with
>   S3 discovery?
>   - Etc.
>
> It seems to me that if we try to fix all this issue, there will be
> virtually no difference between Fabric and Hadoop Accelerator builds except
> couple of scripts and config files. If so, there is no reason to have two
> builds.
>
> -Val
>
> On Mon, Nov 21, 2016 at 3:13 PM, Denis Magda  wrote:
>
> On the separate note, in the Bigtop, we start looking into changing the
>
> way we
>
> deliver Ignite and we'll likely to start offering the whole 'data fabric'
> experience instead of the mere "hadoop-acceleration”.
>
>
> And you still will be using hadoop-accelerator libs of Ignite, right?
>
> I’m thinking of if there is a need to keep releasing Hadoop Accelerator as
> a separate delivery.
> What if we start releasing the accelerator as a part of the standard
> fabric binary putting hadoop-accelerator libs under ‘optional’ folder?
>
> —
> Denis
>
> On Nov 21, 2016, at 12:19 PM, Konstantin Boudnik  wrote:
>
> What Denis said: spark has been added to the Hadoop accelerator as a way
>
> to
>
> boost the performance of more than just MR compute of the Hadoop stack,
>
> IIRC.
>
> For what it worth, Spark is considered a part of Hadoop at large.
>
> On the separate note, in the Bigtop, we start looking into changing the
>
> way we
>
> deliver Ignite and we'll likely to start offering the whole 'data fabric'
> experience instead of the mere "hadoop-acceleration".
>
> Cos
>
> On Mon, Nov 21, 2016 at 09:54AM, Denis Magda wrote:
>
> Val,
>
> Ignite Hadoop module includes not only the map-reduce accelerator but
>
> Ignite
>
> Hadoop File System component as well. The latter can be used in
>
> deployments
>
> like HDFS+IGFS+Ignite Spark + Spark.
>
> Considering this I’m for the second solution proposed by you: put both
>
> 2.10
>
> and 2.11 ignite-spark modules under ‘optional’ folder of Ignite Hadoop
> Accelerator distribution.
> https://issues.apache.org/jira/browse/IGNITE-4254 <
>
> https://issues.apache.org/jira/browse/IGNITE-4254>
>
>
> BTW, this task may be affected or related to the following ones:
> https://issues.apache.org/jira/browse/IGNITE-3596 <
>
> https://issues.apache.org/jira/browse/IGNITE-3596>
>
> https://issues.apache.org/jira/browse/IGNITE-3822
>
> —
> Denis
>
> On Nov 19, 2016, at 1:26 PM, Valentin Kulichenko <
>
> valentin.kuliche...@gmail.com> wrote:
>
>
> Hadoop Accelerator is a plugin to Ignite and this plugin is used by
>
> Hadoop
>
> when running its jobs. ignite-spark module only provides IgniteRDD
>
> which
>
> Hadoop obviously will never use.
>
> Is there another use case for Hadoop Accelerator which I'm missing?
>
> -Val
>
> On Sat, Nov 19, 2016 at 3:12 AM, Dmitriy Setrakyan <
>
> dsetrak...@apache.org>
>
> wrote:
>
> Why do you think that spark module is not needed in our hadoop build?
>
> On Fri, Nov 18, 2016 at 5:44 PM, Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
> Folks,
>
> Is there anyone who understands the