Thanks Dongjoon !
Regards,
Mridul
On Fri, Jun 23, 2023 at 6:58 PM Dongjoon Hyun wrote:
> We are happy to announce the availability of Apache Spark 3.4.1!
>
> Spark 3.4.1 is a maintenance release containing stability fixes. This
> release is based on the branch-3.4 maintenance branch of Spark. W
Thanks for flagging the concern Dongjoon, I was not aware of the discussion
- but I can understand the concern.
Would be great if you or Matei could update the thread on the result of
deliberations, once it reaches a logical consensus: before we set up
official policy around it.
Regards,
Mridul
+CC zhouye...@gmail.com
On Mon, May 23, 2022 at 7:11 AM Han Altae-Tran wrote:
> Hi,
>
> First of all, I am very thankful for all of the amazing work that goes
> into this project! It has opened up so many doors for me! I am a long
> time Spark user, and was very excited to start working with th
Congratulations everyone !
And thanks Gengliang for sheparding the release out :-)
Regards,
Mridul
On Tue, Oct 19, 2021 at 9:25 AM Yuming Wang wrote:
> Congrats and thanks!
>
> On Tue, Oct 19, 2021 at 10:17 PM Gengliang Wang wrote:
>
>> Hi all,
>>
>> Apache Spark 3.2.0 is the third release of
Unfortunate about Mesos, +1 on deprecation of mesos integration.
Regards,
Mridul
On Wed, Apr 7, 2021 at 7:12 AM Sean Owen wrote:
> I noted that Apache Mesos is moving to the attic, so won't be actively
> developed soon:
>
> https://lists.apache.org/thread.html/rab2a820507f7c846e54a847398ab20f4
Thanks Hyukjin and congratulations everyone on the release !
Regards,
Mridul
On Tue, Mar 2, 2021 at 8:54 PM Yuming Wang wrote:
> Great work, Hyukjin!
>
> On Wed, Mar 3, 2021 at 9:50 AM Hyukjin Kwon wrote:
>
>> We are excited to announce Spark 3.1.1 today.
>>
>> Apache Spark 3.1.1 is the second
+1 on pushing the branch cut for increased dev time to match previous
releases.
Regards,
Mridul
On Sat, Oct 3, 2020 at 10:22 PM Xiao Li wrote:
> Thank you for your updates.
>
> Spark 3.0 got released on Jun 18, 2020. If Nov 1st is the target date of
> the 3.1 branch cut, the feature development
Hi,
50% of driver time being spent in gc just for listenerbus sounds very
high in a 30G heap.
Did you try to take a heap dump and see what is occupying so much memory ?
This will help us eliminate if the memory usage is due to some user
code/library holding references to large objects/graph of
Great job everyone ! Congratulations :-)
Regards,
Mridul
On Thu, Jun 18, 2020 at 10:21 AM Reynold Xin wrote:
> Hi all,
>
> Apache Spark 3.0.0 is the first release of the 3.x line. It builds on many
> of the innovations from Spark 2.x, bringing new ideas as well as continuing
> long-term project
If using RDD's, you can use saveAsHadoopFile or saveAsNewAPIHadoopFile
with the conf passed in which overrides the keys you need.
For example, you can do :
val saveConf = new Configuration(sc.hadoopConfiguration)
// configure saveConf with overridden s3 config
rdd.saveAsNewAPIHadoopFile(..., conf
Congratulations, great job everyone !
Regards,
Mridul
On Mon, Aug 15, 2016 at 2:19 PM, Luciano Resende wrote:
> The Apache Bahir PMC is pleased to announce the release of Apache Bahir
> 2.0.0 which is our first major release and provides the following
> extensions for Apache Spark 2.0.0 :
>
> A
Congratulations, great job everyone !
Regards
Mridul
On Monday, August 15, 2016, Luciano Resende wrote:
> The Apache Bahir PMC is pleased to announce the release of Apache Bahir
> 2.0.0 which is our first major release and provides the following
> extensions for Apache Spark 2.0.0 :
>
> Akka S
I think Reynold's suggestion of using ram disk would be a good way to
test if these are the bottlenecks or something else is.
For most practical purposes, pointing local dir to ramdisk should
effectively give you 'similar' performance as shuffling from memory.
Are there concerns with taking that a
We use it in executors to get to :
a) spark conf (for getting to hadoop config in map doing custom
writing of side-files)
b) Shuffle manager (to get shuffle reader)
Not sure if there are alternative ways to get to these.
Regards,
Mridul
On Wed, Mar 16, 2016 at 2:52 PM, Reynold Xin wrote:
> Any
ed, Mar 16, 2016 at 3:29 PM, Mridul Muralidharan
> wrote:
>>
>> b) Shuffle manager (to get shuffle reader)
>
>
> What's the use case for shuffle manager/reader? This seems like using super
> internal APIs in applications.
>
>
-
What I understood from Imran's mail (and what was referenced in his
mail) the RDD mentioned seems to be violating some basic contracts on
how partitions are used in spark [1].
They cannot be arbitrarily numbered,have duplicates, etc.
Extending RDD to add functionality is typically for niche cases
Simply customize your log4j confit instead of modifying code if you don't
want messages from that class.
Regards
Mridul
On Sunday, July 26, 2015, Sea <261810...@qq.com> wrote:
> This exception is so ugly!!! The screen is full of these information when
> the program runs a long time, and they
to LargeByteBuffer, seems
> promising.
>
> thanks,
> Imran
>
> On Tue, Feb 3, 2015 at 7:32 PM, Mridul Muralidharan > wrote:
>
>> That is fairly out of date (we used to run some of our jobs on it ... But
>> that is forked off 1.1 actually).
>>
>> Regard
That is fairly out of date (we used to run some of our jobs on it ... But
that is forked off 1.1 actually).
Regards
Mridul
On Tuesday, February 3, 2015, Imran Rashid wrote:
> Thanks for the explanations, makes sense. For the record looks like this
> was worked on a while back (and maybe the wo
Brilliant stuff ! Congrats all :-)
This is indeed really heartening news !
Regards,
Mridul
On Fri, Oct 10, 2014 at 8:24 PM, Matei Zaharia wrote:
> Hi folks,
>
> I interrupt your regularly scheduled user / dev list to bring you some pretty
> cool news for the project, which is that we've been a
20 matches
Mail list logo