Questions about caching

Greetings, Spark Aficionados- I'm working on a project to (ab-)use PySpark to do particle physics analysis, which involves iterating with a lot of transformations (to apply weights and select candidate events) and reductions (to produce histograms of relevant physics objects). We have a basic

Re: What are the alternatives to nested DataFrames?

Could you join() the DFs on a common key? On Fri, Dec 28, 2018 at 18:35 wrote: > Shabad , I am not sure what you are trying to say. Could you please give > me an example? The result of the Query is a Dataframe that is created after > iterating, so I am not sure how could I map that to a column

Re: Where does the Driver run?

Hi Pat, On Sun, Mar 24, 2019 at 1:03 PM Pat Ferrel wrote: > Thanks, I have seen this many times in my research. Paraphrasing docs: “in > deployMode ‘cluster' the Driver runs on a Worker in the cluster” > > When I look at logs I see 2 executors on the 2 slaves (executor 0 and 1 > with addresses

Re: Where does the Driver run?

Hi Pat, Indeed, I don't think that it's possible to use cluster mode w/o spark-submit. All the docs I see appear to always describe needing to use spark-submit for cluster mode -- it's not even compatible with spark-shell. But it makes sense to me -- if you want Spark to run your application's

Please stop asking to unsubscribe

The correct way to unsubscribe is to mail user-unsubscr...@spark.apache.org Just mailing the list with "unsubscribe" doesn't actually do anything... Thanks Andrew - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Anaconda installation with Pyspark/Pyarrow (2.3.0+) on cloudera managed server

Hi, On Mon, May 6, 2019 at 11:41 AM Patrick McCarthy wrote: > > Thanks Gourav. > > Incidentally, since the regular UDF is row-wise, we could optimize that a bit > by taking the convert() closure and simply making that the UDF. > > Since there's that MGRS object that we have to create too, we

Re: Anaconda installation with Pyspark/Pyarrow (2.3.0+) on cloudera managed server

Hi, On Mon, May 6, 2019 at 11:59 AM Gourav Sengupta wrote: > > Hence, what I mentioned initially does sound correct ? I don't agree at all - we've had a significant boost from moving to regular UDFs to pandas UDFs. YMMV, of course. > > On Mon, May 6, 2019 at 5:43 PM Andrew

Re: Connecting to Spark cluster remotely

Hi Rishkesh On Mon, Apr 22, 2019 at 4:26 PM Rishikesh Gawade wrote: > > To put it simply, what are the configurations that need to be done on the > client machine so that it can run driver on itself and executors on > spark-yarn cluster nodes? TBH, if it were me, I would simply SSH to the

Re: can't download 2.4.1 sourcecode

On Mon, Apr 22, 2019 at 10:54 PM yutaochina wrote: > > >

Re: Driver vs master

ot;local" master and "client mode" then yes tasks execute in the same JVM as the driver". The answer depends on the exact setup Amit has and how the application is configured > HTH... > > Ayan > > > > On Tue, Oct 8, 2019 at 12:11 PM Andrew Melo wrote: > &

Re: Driver vs master

Hi Amit On Mon, Oct 7, 2019 at 18:33 Amit Sharma wrote: > Can you please help me understand this. I believe driver programs runs on > master node If we are running 4 spark job and driver memory config is 4g then total 16 > 6b would be used of master node. This depends on what master/deploy

Re: Driver vs master

a time. I understand that. I think there's a misunderstanding with the terminology, though. Are you running multiple separate spark instances on a single machine or one instance with multiple jobs inside. > > On Monday, October 7, 2019, Andrew Melo wrote: > >> Hi Amit >>

Re: Reading 7z file in spark

It only makes sense if the underlying file is also splittable, and even then, it doesn't really do anything for you if you don't explicitly tell spark about the split boundaries On Tue, Jan 14, 2020 at 7:36 PM Someshwar Kale wrote: > I would suggest to use other compression technique which is

Re: Scala version compatibility

) Thanks for your help, Andrew > On Mon, Apr 6, 2020 at 3:50 PM Andrew Melo wrote: > >> Hello all, >> >> I'm aware that Scala is not binary compatible between revisions. I have >> some Java code whose only Scala dependency is the transitive dependency >> thr

Re: Scala version compatibility

not included in my artifact except for this single callsite. Thanks Andrew > On Mon, Apr 6, 2020 at 4:16 PM Andrew Melo wrote: > >> >> >> On Mon, Apr 6, 2020 at 3:08 PM Koert Kuipers wrote: >> >>> yes it will >>> >>> >> Ooof, I

Scala version compatibility

Hello all, I'm aware that Scala is not binary compatible between revisions. I have some Java code whose only Scala dependency is the transitive dependency through Spark. This code calls a Spark API which returns a Seq, which I then convert into a List with JavaConverters.seqAsJavaListConverter.

Supporting Kryo registration in DSv2

Hello all, Is there a way to register classes within a datasourcev2 implementation in the Kryo serializer? I've attempted the following in both the constructor and static block of my toplevel class: SparkContext context = SparkContext.getOrCreate(); SparkConf conf =

Optimizing LIMIT in DSv2

Hello, Executing "SELECT Muon_Pt FROM rootDF LIMIT 10", where "rootDF" is a temp view backed by a DSv2 reader yields the attached plan [1]. It appears that the initial stage is run over every partition in rootDF, even though each partition has 200k rows (modulo the last partition which holds the

Re: Can I collect Dataset[Row] to driver without converting it to Array [Row]?

Hi Maqy On Wed, Apr 22, 2020 at 3:24 AM maqy <454618...@qq.com> wrote: > > I will traverse this Dataset to convert it to Arrow and send it to Tensorflow > through Socket. (I presume you're using the python tensorflow API, if you're not, please ignore) There is a JIRA/PR ([1] [2]) which

Re: REST Structured Steaming Sink

On Wed, Jul 1, 2020 at 8:13 PM Burak Yavuz wrote: > > I'm not sure having a built-in sink that allows you to DDOS servers is the > best idea either. foreachWriter is typically used for such use cases, not > foreachBatch. It's also pretty hard to guarantee exactly-once, rate limiting, > etc.

PySpark aggregation w/pandas_udf

Hi all, For our use case, we would like to perform an aggregation using a pandas_udf with dataframes that have O(100m) rows and a few 10s of columns. Conceptually, this looks a bit like pyspark.RDD.aggregate, where the user provides: * A "seqOp" which accepts pandas series(*) and outputs an

Re: Spark DataFrame Creation

Hi Mark, On Wed, Jul 22, 2020 at 4:49 PM Mark Bidewell wrote: > > Sorry if this is the wrong place for this. I am trying to debug an issue > with this library: > https://github.com/springml/spark-sftp > > When I attempt to create a dataframe: > > spark.read. >

Re: Merge two dataframes

t merge join Cheers Andrew On Wed, May 12, 2021 at 11:32 AM Andrew Melo wrote: > > Hi, > > In the case where the left and right hand side share a common parent like: > > df = spark.read.someDataframe().withColumn('rownum', row_number()) > df1 = df.withColumn('c1', expensive_

Re: Merge two dataframes

ch. A big goal of mine is to make it so that what was changed is recomputed, and no more, which will speed up the rate at which we can find new physics. Cheers Andrew > > On 5/17/21, 2:56 PM, "Andrew Melo" wrote: > > CAUTION: This email originated from outside of t

Re: Merge two dataframes

explicitly compute them themselves. Cheers Andrew On Mon, May 17, 2021 at 1:10 PM Sean Owen wrote: > > Why join here - just add two columns to the DataFrame directly? > > On Mon, May 17, 2021 at 1:04 PM Andrew Melo wrote: >> >> Anyone have ideas about the below Q? >> &g

Re: Merge two dataframes

Hi, In the case where the left and right hand side share a common parent like: df = spark.read.someDataframe().withColumn('rownum', row_number()) df1 = df.withColumn('c1', expensive_udf1('foo')).select('c1', 'rownum') df2 = df.withColumn('c2', expensive_udf2('bar')).select('c2', 'rownum')

Grabbing the current MemoryManager in a plugin

Hello, I've implemented support for my DSv2 plugin to back its storage with ArrowColumnVectors, which necessarily means using off-heap memory. Is it possible to somehow grab either a reference to the current MemoryManager so that the off-heap memory usage is properly accounted for and to prevent

Re: Grabbing the current MemoryManager in a plugin

Hello, Any wisdom on the question below? Thanks Andrew On Fri, Apr 8, 2022 at 16:04 Andrew Melo wrote: > Hello, > > I've implemented support for my DSv2 plugin to back its storage with > ArrowColumnVectors, which necessarily means using off-heap memory. Is > it possible to some

Re: cannot access class sun.nio.ch.DirectBuffer

Hi Sean, Out of curiosity, will Java 11+ always require special flags to access the unsafe direct memory interfaces, or is this something that will either be addressed by the spec (by making an "approved" interface) or by libraries (with some other workaround)? Thanks Andrew On Tue, Apr 12,

Re: cannot access class sun.nio.ch.DirectBuffer

. > > On Wed, Apr 13, 2022 at 9:05 AM Andrew Melo wrote: > >> Hi Sean, >> >> Out of curiosity, will Java 11+ always require special flags to access >> the unsafe direct memory interfaces, or is this something that will either >> be addressed by the spec (by

Re: How is union() implemented? Need to implement column bind

It would certainly be useful for our domain to have some sort of native cbind(). Is there a fundamental disapproval of adding that functionality, or is it just a matter of nobody implementing it? On Wed, Apr 20, 2022 at 16:28 Sean Owen wrote: > Good lead, pandas on Spark concat() is worth

Re: [EXTERNAL] RDD.pipe() for binary data

I'm curious about using shared memory to speed up the JVM->Python round trip. Is there any sane way to do anonymous shared memory in Java/scale? On Sat, Jul 16, 2022 at 16:10 Sebastian Piu wrote: > Other alternatives are to look at how PythonRDD does it in spark, you > could also try to go for

PySpark cores

Hello, Is there a way to tell Spark that PySpark (arrow) functions use multiple cores? If we have an executor with 8 cores, we would like to have a single PySpark function use all 8 cores instead of having 8 single core python functions run. Thanks! Andrew

Re: Profiling PySpark Pandas UDF

Hi Gourav, Since Koalas needs the same round-trip to/from JVM and Python, I expect that the performance should be nearly the same for UDFs in either API Cheers Andrew On Thu, Aug 25, 2022 at 11:22 AM Gourav Sengupta wrote: > > Hi, > > May be I am jumping to conclusions and making stupid

Re: [Spark Sql] Global Setting for Case-Insensitive String Compare

I think this is the right place, just a hard question :) As far as I know, there's no "case insensitive flag", so YMMV On Mon, Nov 21, 2022 at 5:40 PM Patrick Tucci wrote: > > Is this the wrong list for this type of question? > > On 2022/11/12 16:34:48 Patrick Tucci wrote: > > Hello, > > > >