Re: Spark 3.0 yarn does not support cdh5

2019-10-21 Thread melin li 
Many clusters still use cdh5, and want to continue to support cdh5,cdh5
based on hadoop 2.6

melin li  于2019年10月21日周一 下午3:02写道：

> 很多集群还是使用cdh5，希望继续支持cdh5，cdh5是基于hadoop 2.6
>
> dev/make-distribution.sh --tgz -Pkubernetes -Pyarn -Phive-thriftserver
> -Phive -Dhadoop.version=2.6.0-cdh5.15.0 -DskipTest
>
> ```
> [INFO] Compiling 25 Scala sources to
> /Users/libinsong/Documents/codes/tongdun/spark-3.0/resource-managers/yarn/target/scala-2.12/classes
> ...
> [ERROR] [Error]
> /Users/libinsong/Documents/codes/tongdun/spark-3.0/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:298:
> value setRolledLogsIncludePattern is not a member of
> org.apache.hadoop.yarn.api.records.LogAggregationContext
> [ERROR] [Error]
> /Users/libinsong/Documents/codes/tongdun/spark-3.0/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:300:
> value setRolledLogsExcludePattern is not a member of
> org.apache.hadoop.yarn.api.records.LogAggregationContext
> [ERROR] [Error]
> /Users/libinsong/Documents/codes/tongdun/spark-3.0/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:551:
> not found: value isLocalUri
> [ERROR] [Error]
> /Users/libinsong/Documents/codes/tongdun/spark-3.0/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:1367:
> not found: value isLocalUri
> [ERROR] four errors found
> ```
>

Spark 3.0 yarn does not support cdh5

2019-10-21 Thread melin li 
很多集群还是使用cdh5，希望继续支持cdh5，cdh5是基于hadoop 2.6

dev/make-distribution.sh --tgz -Pkubernetes -Pyarn -Phive-thriftserver
-Phive -Dhadoop.version=2.6.0-cdh5.15.0 -DskipTest

```
[INFO] Compiling 25 Scala sources to
/Users/libinsong/Documents/codes/tongdun/spark-3.0/resource-managers/yarn/target/scala-2.12/classes
...
[ERROR] [Error]
/Users/libinsong/Documents/codes/tongdun/spark-3.0/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:298:
value setRolledLogsIncludePattern is not a member of
org.apache.hadoop.yarn.api.records.LogAggregationContext
[ERROR] [Error]
/Users/libinsong/Documents/codes/tongdun/spark-3.0/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:300:
value setRolledLogsExcludePattern is not a member of
org.apache.hadoop.yarn.api.records.LogAggregationContext
[ERROR] [Error]
/Users/libinsong/Documents/codes/tongdun/spark-3.0/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:551:
not found: value isLocalUri
[ERROR] [Error]
/Users/libinsong/Documents/codes/tongdun/spark-3.0/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:1367:
not found: value isLocalUri
[ERROR] four errors found
```

Orc upgrade to 1.6 to support zstd

2019-10-18 Thread melin li 
Orc upgrade to 1.6 to support zstd
https://issues.apache.org/jira/browse/ORC-363