[
https://issues.apache.org/jira/browse/HUDI-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ethan Guo closed HUDI-2747.
---------------------------
Resolution: Fixed
> Fix hudi cli metadata commands
> ------------------------------
>
> Key: HUDI-2747
> URL: https://issues.apache.org/jira/browse/HUDI-2747
> Project: Apache Hudi
> Issue Type: Bug
> Affects Versions: 0.10.0
> Reporter: sivabalan narayanan
> Assignee: Yue Zhang
> Priority: Blocker
> Labels: HUDI-bug, pull-request-available
> Fix For: 0.11.0
>
>
> Fix hudi cli metadata commands. Currently when running hudi cli metadata
> commands locally, it complains about spark master, e.g., yarn master not
> found. The commands need to take in spark related configs so that they can
> run locally and in cluster.
> {code:java}
> java.lang.reflect.UndeclaredThrowableException
> at
> org.springframework.util.ReflectionUtils.rethrowRuntimeException(ReflectionUtils.java:315)
> at
> org.springframework.util.ReflectionUtils.handleInvocationTargetException(ReflectionUtils.java:295)
> at
> org.springframework.util.ReflectionUtils.handleReflectionException(ReflectionUtils.java:279)
> at
> org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:219)
> at
> org.springframework.shell.core.SimpleExecutionStrategy.invoke(SimpleExecutionStrategy.java:68)
> at
> org.springframework.shell.core.SimpleExecutionStrategy.execute(SimpleExecutionStrategy.java:59)
> at
> org.springframework.shell.core.AbstractShell.executeCommand(AbstractShell.java:134)
> at
> org.springframework.shell.core.JLineShell.promptLoop(JLineShell.java:533)
> at org.springframework.shell.core.JLineShell.run(JLineShell.java:179)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.spark.SparkException: Could not parse Master URL: 'yarn'
> at
> org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2784)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
> at
> org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
> at
> org.apache.hudi.cli.utils.SparkUtil.initJavaSparkConf(SparkUtil.java:117)
> at
> org.apache.hudi.cli.commands.MetadataCommand.initJavaSparkContext(MetadataCommand.java:362)
> at
> org.apache.hudi.cli.commands.MetadataCommand.listPartitions(MetadataCommand.java:196)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:216)
> ... 6 more {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
