Re: Please add me as a member of Apache Mnemonic in Github if appropriate

[Gary]

Thanks Henry for the explanation, Yes, That way would be great, I'm
looking forward to this new feature :)

On 3/30/2016 5:45 PM, Henry Saputra wrote:
> No, the Github repository for Apache Mnemonic is only a mirror to ASF git
> repository [1].
> So you can not accept pull requests from github.
>
> But we could ask for advance integration with github so we when a merge to
> ASF Git happen with comment This closes # it would automatically
> close the PR.
>
> - Henry
>
> [1] https://git-wip-us.apache.org/repos/asf?p=incubator-mnemonic
>
> On Wed, Mar 30, 2016 at 5:31 PM, Zheng, Kai  wrote:
>
>> There must be some setup to go with the PR push/pull process. Currently
>> it's still read-only. Please fire a INFRA jira for this.
>>
>> Regards,
>> Kai
>>
>> -Original Message-
>> From: Gary [mailto:ga...@apache.org]
>> Sent: Thursday, March 31, 2016 8:27 AM
>> To: dev@mnemonic.incubator.apache.org
>> Subject: Please add me as a member of Apache Mnemonic in Github if
>> appropriate
>>
>> I found I cannot accept pull request in Github so should I request to
>> become a member of Apache Mnemonic in Github ? or what I can do for this
>> case ? Thanks.
>>
>>




signature.asc
Description: OpenPGP digital signature

[jira] [Commented] (MNEMONIC-3) set up http://mnemonic.incubator.apache.org web page

[Wang, Gang (JIRA)]

[ 
https://issues.apache.org/jira/browse/MNEMONIC-3?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219135#comment-15219135
 ] 

Wang, Gang commented on MNEMONIC-3:
---

Thanks Henry. Hello Jekyll, one more new tech. for me to learn :)

> set up http://mnemonic.incubator.apache.org web page
> 
>
> Key: MNEMONIC-3
> URL: https://issues.apache.org/jira/browse/MNEMONIC-3
> Project: Mnemonic
>  Issue Type: Improvement
>Reporter: Yanping Wang
>Assignee: Debo Dutta
>
> After source code move into  Git 
> https://git-wip-us.apache.org/repos/asf/incubator-mnemonic.git  and JIRA set 
> at https://issues.apache.org/jira/browse/MNEMONIC  we shall build 
> http://mnemonic.incubator.apache.org.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (MNEMONIC-3) set up http://mnemonic.incubator.apache.org web page

[Wang, Gang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MNEMONIC-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wang, Gang updated MNEMONIC-3:
--
Comment: was deleted

(was: Thanks Henry. Hello Jekyll, one more new tech. for me to learn :))

> set up http://mnemonic.incubator.apache.org web page
> 
>
> Key: MNEMONIC-3
> URL: https://issues.apache.org/jira/browse/MNEMONIC-3
> Project: Mnemonic
>  Issue Type: Improvement
>Reporter: Yanping Wang
>Assignee: Debo Dutta
>
> After source code move into  Git 
> https://git-wip-us.apache.org/repos/asf/incubator-mnemonic.git  and JIRA set 
> at https://issues.apache.org/jira/browse/MNEMONIC  we shall build 
> http://mnemonic.incubator.apache.org.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

RE: Please add me as a member of Apache Mnemonic in Github if appropriate

[Zheng, Kai]

There must be some setup to go with the PR push/pull process. Currently it's 
still read-only. Please fire a INFRA jira for this.

Regards,
Kai

-Original Message-
From: Gary [mailto:ga...@apache.org] 
Sent: Thursday, March 31, 2016 8:27 AM
To: dev@mnemonic.incubator.apache.org
Subject: Please add me as a member of Apache Mnemonic in Github if appropriate

I found I cannot accept pull request in Github so should I request to become a 
member of Apache Mnemonic in Github ? or what I can do for this case ? Thanks.

[jira] [Commented] (MNEMONIC-3) set up http://mnemonic.incubator.apache.org web page

[Wang, Gang (JIRA)]

[ 
https://issues.apache.org/jira/browse/MNEMONIC-3?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219090#comment-15219090
 ] 

Wang, Gang commented on MNEMONIC-3:
---

Apache Flink seems using HTML/CSS/JS tech. to construct their website so 
basically it is a traditional static website that probably also fit into our 
case.

> set up http://mnemonic.incubator.apache.org web page
> 
>
> Key: MNEMONIC-3
> URL: https://issues.apache.org/jira/browse/MNEMONIC-3
> Project: Mnemonic
>  Issue Type: Improvement
>Reporter: Yanping Wang
>Assignee: Debo Dutta
>
> After source code move into  Git 
> https://git-wip-us.apache.org/repos/asf/incubator-mnemonic.git  and JIRA set 
> at https://issues.apache.org/jira/browse/MNEMONIC  we shall build 
> http://mnemonic.incubator.apache.org.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: A Proposal Apache Incubator Mnemonic as an alternative infra. for Apache Arrow

[Wang, Yanping]

Hi, All

I met with Jacques today at Strata, we think it would be great that Arrow and 
Mnemonic communities can have a F2F meeting together to talk about our 
integration.
I have following two days, 4/11 Monday afternoon, or 4/15 Friday.
We can meet at  intel SC campus.

Would you let me know if you are able to join us and which day you'd prefer?

Thanks
Yanping


On Mar 29, 2016, at 4:38 PM, Gary > 
wrote:

Yes, I agree with you and that's great if we could brainstorm here to collect 
more ideas about enabling non-volatile memory usage for Apache Arrow through 
Mnemonic.

for the questions, my ideas are:


- Right now you are using unpooled persistent memory. Does that make sense or 
does chunking make more sense?

Gary: I think it could make some sense if developer knows that their datasets 
are very big and they want Apache Arrow to keep most of them in memory for 
intensive computing e.g. sort.
  the developer certainly can spill their Mnemonic managed datasets 
into disk but this way seems a bit inefficient in some scenarios that might 
depend on concrete application logic .


- What do you think is the right way to transition back and forth between 
persistent and ephemeral memory? What do you think will be the first pattern to 
be adopted. For example, do you think we should try to use it as a tiered 
storage for sort spilling (before hitting the disk), or should we use it for 
caching?
Gary: my 2 cents, the netty library looks not yet provide a elegant switch 
mechanism for Arrow to use, probably we can change the logic around 
"initialCapacity > directArena.chunkSize" to control which buffer put on 
off-heap or managed by Mnemonic, another approach is to let memory clustering 
mechanism of Mnemonic managing hybrid memory-like spaces instead of part logics 
of class PooledByteBufAllocatorL.
Regarding the sorting, I think it is a typical case of random access to the 
data, we should avoid spilling as much as possible.
my 2 cents, the performance could be
all in off-heap if possible > mnemonic used as cache > all in mnemonic using 
NVMe/disk >  off-heap + spilling
the code simplicity would be
all in off-heap if possible >  all in mnemonic using NVMe/disk > mnemonic used 
as cache >  off-heap + spilling

the reason why the mode "mnemonic used as cache + spilling" probably 
unnecessary is mnemonic could provide nearly equivalent capacity of disk.

Thanks.
Gary.


-Original Message-

From: Jacques Nadeau [mailto:jacq...@apache.org]

Sent: Tuesday, March 29, 2016 8:05 AM

To:  
d...@arrow.apache.org

Subject: Re: A Proposal Apache Incubator Mnemonic as an alternative infra. for 
Apache Arrow



This is super cool. A couple of questions:



- Right now you are using unpooled persistent memory. Does that make sense or 
does chunking make more sense?

- What do you think is the right way to transition back and forth between 
persistent and ephemeral memory? What do you think will be the first pattern to 
be adopted. For example, do you think we should try to use it as a tiered 
storage for sort spilling (before hitting the disk), or should we use it for 
caching?



I think it will be much easier to think about this in the context of a primary 
or first use case. Do you have something in mind or should we brainstorm here?



On Wed, Mar 23, 2016 at 7:16 PM, Gary 
> wrote:



> Hello,

>

>We have created a patch for Apache Arrow to leverage Apache

> incubator Mnemonic as an alternative infra. for underlying memory

> resources allocation, you can find it as below forked repo.

>

>  
> https://github.com/NonVolatileComputing/arrow

>

> By this way, Apache Arrow could take some structural benefits from

> Mnemonic project they are

>

> - Arrow is able to leverage larger capacity of high performance

> hybrid storage devices. e.g. high-end SSD, NVMe

>

> - Mnemonic provide a potential opportunity for Arrow to

> optimize/tuning its allocation algorithms as a native Arrow-oriented

> allocation services

>

> - The non-volatile features of  Mnemonic make it possible that

> Arrow could make its columnar in-memory data shared between different

> applications or across life-cycle of single application

>

> - Arrow could take advantages of coming Mnemonic features of

> memory clustering/DOG (distributed object graph) and massive native

> computing

>

> - Mnemonic helps to reduce the pressure of main memory utilization

> and its related system wide overheads.

>

>Our this patch is designed to minimize the changes for user to use

> Arrow, please check out the test cases provided by this patch for your

> reference.

>

>Note that, we need to put allocator services to a specified

> position (indicated by pom.xml) for Mnemonic backed Arrow related test

> cases to 

Mnemonic board report due in one week.

[Patrick Hunt]

Hi folks. The Mnemonic board report is due in one week, Weds April
6th. You will be reporting monthly for the first quarter, then
quarterly after that. The final form of the report needs to be posted
here by April 6th:
http://wiki.apache.org/incubator/April2016

Most communities put together a draft report following the format laid
out on that link in their public dev list. Then post it to the wiki.
If you have any questions please ask and the mentors can provide
insight.

Patrick