t
> > process or the store).
> >
> > Can you check if the object store evicting objects (it prints something
> to
> > stdout/stderr when this happens)? Could you be running out of memory but
> > failing to release the objects?
> >
> > On Tue, Jul 10, 201
art the
Jupyter kernel.
On Tue, Jul 10, 2018 at 12:27 PM Corey Nolet wrote:
> Wes,
>
> Unfortunately, my code is on a separate network. I'll try to explain what
> I'm doing and if you need further detail, I can certainly pseudocode
> specifics.
>
> I am using multiproce
gure out where and what I'm doing.
Thanks again!
On Tue, Jul 10, 2018 at 12:05 PM Wes McKinney wrote:
> hi Corey,
>
> Can you provide the code (or a simplified version thereof) that shows
> how you're using Plasma?
>
> - Wes
>
> On Tue, Jul 10, 2018 at 11:45 AM,
I'm on a system with 12TB of memory and attempting to use Pyarrow's Plasma
client to convert a series of CSV files (via Pandas) into a Parquet store.
I've got a little over 20k CSV files to process which are about 1-2gb each.
I'm loading 500 to 1000 files at a time.
In each iteration, I'm loading
lient object successfully (it has
> a socket connection to the store).
>
> On Wed, May 16, 2018 at 3:43 PM Corey Nolet wrote:
>
>> Robert,
>>
>> Thank you for the quick response. I've been playing around for a few hours
>> to get a feel for how this
think of it as a replacement for
> Python multiprocessing that automatically uses shared memory and Arrow for
> serialization.
>
> On Wed, May 16, 2018 at 10:02 AM Corey Nolet wrote:
>
> > I've been reading through the PyArrow documentation and trying to
> > understand
I've been reading through the PyArrow documentation and trying to
understand how to use the tool effectively for IPC (using zero-copy).
I'm on a system with 586 cores & 1TB of ram. I'm using Panda's Dataframes
to process several 10's of gigs of data in memory and the pickling that is
done by Pytho
On the Accumulo project, we were getting spam up until 2 hours ago. We were
still using hadoops permissions scheme. I think k the lockdown only works
if you are using the default permissions scheme. Once we flipped to
default, the spam stopped
On Apr 22, 2016 3:11 PM, "Zheng, Kai" wrote:
>
> Fr
Nevermind, I just noticed the message from infrastructure. Looks like it
affected a bunch of projects.
On Thu, Apr 21, 2016 at 10:07 PM, Corey Nolet wrote:
> Should someone request that this account is suspended until further
> notice? It looks like this person may have been compromised.
Should someone request that this account is suspended until further notice?
It looks like this person may have been compromised.
On Thu, Apr 21, 2016 at 7:08 PM, Daniel Takamori (JIRA)
wrote:
>
> [
> https://issues.apache.org/jira/browse/ARROW-117?page=com.atlassian.jira.plugin.system.issue
t in current Linux
> >> (work
> >>> going on currently to address this). So you're left with hugetlbfs,
> which
> >>> involves static allocations and much more pain.
> >>>
> >>> All the above is a long way to say: let's make sure
I was seeing Netty's unsafe classes being used here, not mapped byte
buffer not sure if that statement is completely correct but I'll have to
dog through the code again to figure that out.
The more I was looking at unsafe, it makes sense why that would be
used.apparently it's also supposed to be
Myself as well, thanks!
On Fri, Mar 11, 2016 at 3:26 PM, pino patera wrote:
> Would like an invite! Thx
> On Fri, 11 Mar 2016 at 21:19, Dale Jung wrote:
>
> > Would love an invite as well. Thanks!
> >
> > On Fri, Mar 11, 2016 at 3:15 PM, Prateek Rungta
> > wrote:
> >
> > > Could I get an invi
Russel,
Please send a message to dev-unsubscr...@arrow.apache.org
On Wed, Feb 24, 2016 at 2:56 PM, Russell Simmons <
russell.emergen...@gmail.com> wrote:
> thx
>
t other options do you have in mind?
>
> On Wed, Feb 24, 2016 at 11:30 AM Corey Nolet wrote:
>
> > Agreed,
> >
> > I thought the whole purpose was to share the memory space (using possibly
> > unsafe operations like ByteBuffers) so that it could be directly shared
.
> >
> > IIUC, with Arrow, when application A shares data with application B, the
> > data is still duplicated in the memory spaces of A and B. It's just that
> > data serialization/deserialization are much faster with Arrow (compared
> > with Protobuf).
> >
> >
with Arrow, when application A shares data with application B, the
> data is still duplicated in the memory spaces of A and B. It's just that
> data serialization/deserialization are much faster with Arrow (compared
> with Protobuf).
>
> On Wed, Feb 24, 2016 at 10:40 AM Corey
Forgive me if this question seems ill-informed. I just started looking at
Arrow yesterday. I looked around the github a tad.
Are you expecting the memory space held by one application to be mutable by
that application and made available to all applications trying to read the
memory space?
18 matches
Mail list logo
