Pyarrow version 0.12.1, arrow jar version 0.10, can run correctly. Pyarrow
version 0.121, arrow jar version 0.12, this exception occurs:
> *Expected schema message in stream, was null or length 0*
Pyarrow was upgraded to version 0.12.1 by other package dependency,
resulting in inconsistency betw
https://issues.apache.org/jira/browse/HIVE-13632
李斌松 于2018年12月29日周六 下午4:08写道:
> Hive has fixed this problem, which is not fixed in
> hive-exec-1.2.1.spark2.jar
>
> [image: image.png]
>
>
Import parquet-hadoop-bundle jar. into the spark hive project When you
compress data using zstd, you may load it preferentially from the
parquet-hadoop-bundle, and you canundefinedt find the enum constant
parquet.hadoop.metadata.CompressionCodecName.ZSTD
>
> 18/12/20 10:35:28 ERROR Executor: Excep
If there is timestamp type data in DF, Spark 2.3 toPandas is much slower
than spark 2.2.
Hi, sparks:
I'm using spark2.3 and had found a bug in spark dataframe, here is my
codes:
sc = sparkSession.sparkContext
tmp = sparkSession.createDataFrame(sc.parallelize([[1, 2, 3, 4],
[1, 2, 5, 6], [2, 3, 4, 5], [2, 3, 5, 6]])).toDF('a', 'b', 'c', 'd')
tmp.createOrRep
For example, the user's bank card number cannot be viewed by an analyst and
replaced by an asterisk. How do you do that in spark?
Limit the number of tasks submitted to avoid a task occupancy attitude
resources, while you can guide users to set reasonable conditions,
[image: 内嵌图片 1]
spark_submit_tasks_threshold.patch
Description: Binary data
-
To unsubscr
Does spark support hive table(parquet) column renaming?
Custom function cannot be accessed across database,
example: The registration function json_extract_value is in database A, and
A.json_extract_value cannot be called in the database B
SessionCatalog.java
externalCatalog.getFunction(currentDb, name.funcName)
to
externalCatalog.getFunction(name.d
Create a temporary function, reference HDFS on the jar file, update the jar
file, not immediately effective, need to restart hiveserver
Spark thriff server hiveStatement.getQueryLog return empty?
Through the JDBC connection spark thriftserver, execte hive SQL, check
whether the table read or write permission to expand hook in hive on spark,
you can control permissions, spark on hive what is the point of expansion?
Spark read hive table, catalog. CurrentDatabase value is the default, how
the sparksession initialization, set currentDatabase value?
hive.metastore.uris
thrift://localhost:9083
IP address (or fully-qualified domain name) and
port of the metastore host
What is the difference between hive on spark and spark on hive?
The spark hive udf can read broadcast the variables?
How to reflect dynamic registration udf?
java.lang.UnsupportedOperationException: Schema for type _$13 is not
supported
at
org.apache.spark.sql.catalyst.ScalaReflection$class.schemaFor(ScalaReflection.scala:153)
at
org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:29)
16 matches
Mail list logo