Re: Compute pairwise distance

Hi Debasish, All, I see the status of SPARK-4823 [0] is "in-progress" still. I couldn't gather from the relevant pull request [1] if part of it is already in 1.6.0 (it's closed now). We are facing the same problem of computing pairwise distances between vectors where rows are > 5M and columns in

Re: reading the parquet file in spark sql

>From the parquet file content (dir content) it doesn't look like that parquet write was successful or complete. On Mon, Mar 7, 2016 at 11:17 AM, Angel Angel wrote: > Hello Sir/Madam, > > I am running one spark application having 3 slaves and one master. > > I am wring

Re: AM creation in yarn client mode

My pardon to writing that "there is no AM". I realize it! :-) :-) On Wed, Feb 10, 2016 at 7:14 PM, Steve Loughran <ste...@hortonworks.com> wrote: > > On 10 Feb 2016, at 13:20, Manoj Awasthi <awasthi.ma...@gmail.com> wrote: > > > > On Wed, Feb

Re: AM creation in yarn client mode

On Wed, Feb 10, 2016 at 5:20 PM, Steve Loughran wrote: > > On 10 Feb 2016, at 04:42, praveen S wrote: > > Hi, > > I have 2 questions when running the spark jobs on yarn in client mode : > > 1) Where is the AM(application master) created : > > > in

Re: Client versus cluster mode

The only difference is that in yarn-cluster mode your driver runs within a yarn container (called AM or application master). You would want to run your production jobs in yarn-cluster mode while for development environment may do with yarn-client mode. Again, I think this just a recommendation

Re: How to group multiple row data ?

Sorry but I didn't fully understand the grouping. This line: The group must only take the closest previous trigger. The first one hence shows alone. Can you please explain further? On Wed, Apr 29, 2015 at 4:42 PM, bipin bipin@gmail.com wrote: Hi, I have a ddf with schema (CustomerID,