Re: Code fails when AQE enabled in Spark 3.1

it looks that this commit ( https://github.com/apache/spark/commit/a85490659f45410be3588c669248dc4f534d2a71) do the trick. [image: image.png] Don't you think, this bug is enough important to incluide in 3.1 branch? Regards El jue, 20 ene 2022 a las 8:55, Gaspar Muñoz () escribió: >

Code fails when AQE enabled in Spark 3.1

n be found here: https://issues.apache.org/jira/browse/SPARK-37898 Regards -- Gaspar Muñoz Soria

Re: Spark SQL - Truncate Day / Hour

s not using any open library for handling dates i.e > apache-common , Why implementing it instead of reusing open source? > > Thanks David > -- Gaspar Muñoz Soria Vía de las dos Castillas, 33, Ática 4, 3ª Planta 28224 Pozuelo de Alarcón, Madrid Tel: +34 91 828 6473

Re: spark-avro aliases incompatible

h still exists because of backward > compatibility. > > Please refer to the following documentation to take advantage of the > recent changes in SPARK: https://docs.databricks.com/spark/latest/ > data-sources/read-avro.html > > Kindly let us know how things are going on. > > Reg

Re: spark-avro aliases incompatible

tails regarding the environment, versions, > libraries and code snippets please? > > For example: SPARK version, OS, distribution, running on YARN, etc and all > other details. > > > Regards, > Gourav Sengupta > > On Sun, Nov 5, 2017 at 9:03 AM, Gaspar Muñoz wrote

spark-avro aliases incompatible

ad in spark-avro library "At the moment, it ignores docs, aliases and other properties present in the Avro file". How do you manage aliases and column renaming? Is there any workaround? Thanks in advance. Regards -- Gaspar Muñoz Soria Vía de las dos Castillas, 33, Ática 4, 3ª Planta 2

Re: Best way to merge final output part files created by Spark job

pache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > > -- Gaspar Muñoz @gmunozsoria <http://www.stratio.co

Pagination on big table, splitting joins

Do the processing ) What do you think? Regards. -- Gaspar Muñoz @gmunozsoria <http://www.stratio.com/> Vía de las dos Castillas, 33, Ática 4, 3ª Planta 28224 Pozuelo de Alarcón, Madrid Tel: +34 91 352 59 42 // *@stratiobd <https://twitter.com/StratioBD>*

Re: reduceByKey

0,1,0,0* > > I want to get : > *2553: 0,0,0,1,0,1,0,0* > 46551: 0,1,0,0,0,0,0,0 > 266: 0,1,0,0,0,0,0,0 > *225546: 0,0,0,0,0,2,0,0* > > Anyone can help me getting that? > Thank you. > > Have a nice day. > yasemin > > -- > hiç ender hiç > -- Gaspar Muñoz

Re: Spark Mongodb connection

from Mongodb database. I want to learn > how can I connect Mongodb database to my project. Any one can help for this > issue. > > Have a nice day > yasemin > > -- > hiç ender hiç > -- Gaspar Muñoz @gmunozsoria <http://www.stratio.com/> Vía de las dos Casti

Re: java.lang.IncompatibleClassChangeError when using PrunedFilteredScan

> MongodbRelation( > > MongodbConfigBuilder() > > .set(Host,host) > > .set(Database,database) > > .set(Collection,collection) > > .set(SamplingRatio,samplingRatio).build())(sqlContext) > > > > } > > > > } > > > > In createRelation function it returns MongodbRelation, which is extended > from PrunedFilteredScan > > case class MongodbRelation( > > config: DeepConfig, > > schemaProvided: Option[StructType] = None)( > > @transient val sqlContext: SQLContext) extends PrunedFilteredScan { > > > > Since both TableScan and PrunedFilteredScan are based on BaseRelation, I’m > not sure why clazz.newInstance() call failed by > java.lang.IncompatibleClassChangeError. > > Is there any special way to deal with if I need to use PrunedFilteredScan? > > > > I’m using scala 2.10 and JDK 1.7 > > > > Thanks > > TW > > > -- Gaspar Muñoz @gmunozsoria <http://www.stratio.com/> Vía de las dos Castillas, 33, Ática 4, 3ª Planta 28224 Pozuelo de Alarcón, Madrid Tel: +34 91 352 59 42 // *@stratiobd <https://twitter.com/StratioBD>*