[
https://issues.apache.org/jira/browse/HUDI-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ethan Guo updated HUDI-2747:
----------------------------
Description:
Fix hudi cli metadata commands. Currently when running hudi cli metadata
commands locally, it complains about spark master, e.g., yarn master not found.
The commands need to take in spark related configs so that they can run
locally and in cluster.
{code:java}
java.lang.reflect.UndeclaredThrowableException
at
org.springframework.util.ReflectionUtils.rethrowRuntimeException(ReflectionUtils.java:315)
at
org.springframework.util.ReflectionUtils.handleInvocationTargetException(ReflectionUtils.java:295)
at
org.springframework.util.ReflectionUtils.handleReflectionException(ReflectionUtils.java:279)
at
org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:219)
at
org.springframework.shell.core.SimpleExecutionStrategy.invoke(SimpleExecutionStrategy.java:68)
at
org.springframework.shell.core.SimpleExecutionStrategy.execute(SimpleExecutionStrategy.java:59)
at
org.springframework.shell.core.AbstractShell.executeCommand(AbstractShell.java:134)
at
org.springframework.shell.core.JLineShell.promptLoop(JLineShell.java:533)
at org.springframework.shell.core.JLineShell.run(JLineShell.java:179)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.spark.SparkException: Could not parse Master URL: 'yarn'
at
org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2784)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
at
org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at
org.apache.hudi.cli.utils.SparkUtil.initJavaSparkConf(SparkUtil.java:117)
at
org.apache.hudi.cli.commands.MetadataCommand.initJavaSparkContext(MetadataCommand.java:362)
at
org.apache.hudi.cli.commands.MetadataCommand.listPartitions(MetadataCommand.java:196)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:216)
... 6 more {code}
was:Fix hudi cli metadata commands. Currently when running hudi cli metadata
commands locally, it complains about spark master, e.g., yarn master not found.
The commands need to take in spark related configs so that they can run
locally and in cluster.
> Fix hudi cli metadata commands
> ------------------------------
>
> Key: HUDI-2747
> URL: https://issues.apache.org/jira/browse/HUDI-2747
> Project: Apache Hudi
> Issue Type: Bug
> Affects Versions: 0.10.0
> Reporter: sivabalan narayanan
> Assignee: Yue Zhang
> Priority: Blocker
> Labels: HUDI-bug
> Fix For: 0.11.0
>
>
> Fix hudi cli metadata commands. Currently when running hudi cli metadata
> commands locally, it complains about spark master, e.g., yarn master not
> found. The commands need to take in spark related configs so that they can
> run locally and in cluster.
> {code:java}
> java.lang.reflect.UndeclaredThrowableException
> at
> org.springframework.util.ReflectionUtils.rethrowRuntimeException(ReflectionUtils.java:315)
> at
> org.springframework.util.ReflectionUtils.handleInvocationTargetException(ReflectionUtils.java:295)
> at
> org.springframework.util.ReflectionUtils.handleReflectionException(ReflectionUtils.java:279)
> at
> org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:219)
> at
> org.springframework.shell.core.SimpleExecutionStrategy.invoke(SimpleExecutionStrategy.java:68)
> at
> org.springframework.shell.core.SimpleExecutionStrategy.execute(SimpleExecutionStrategy.java:59)
> at
> org.springframework.shell.core.AbstractShell.executeCommand(AbstractShell.java:134)
> at
> org.springframework.shell.core.JLineShell.promptLoop(JLineShell.java:533)
> at org.springframework.shell.core.JLineShell.run(JLineShell.java:179)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.spark.SparkException: Could not parse Master URL: 'yarn'
> at
> org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2784)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
> at
> org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
> at
> org.apache.hudi.cli.utils.SparkUtil.initJavaSparkConf(SparkUtil.java:117)
> at
> org.apache.hudi.cli.commands.MetadataCommand.initJavaSparkContext(MetadataCommand.java:362)
> at
> org.apache.hudi.cli.commands.MetadataCommand.listPartitions(MetadataCommand.java:196)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:216)
> ... 6 more {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
