[
https://issues.apache.org/jira/browse/GOBBLIN-1055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17039683#comment-17039683
]
Chris Li edited comment on GOBBLIN-1055 at 2/19/20 3:52 AM:
------------------------------------------------------------
{quote}{quote}[Loaded *org.apache.gobblin.util.WritableShimSerialization* from
file:/export/apps/azkaban/azkaban-exec-server/azkaban-exec-server-1102356.01216_00358/executions/22966290/downloadUnzipDir/lib/gobblin-multistage-0.0.252-SNAPSHOT-all.jar]
07-02-2020 18:10:18 PST gobblin-ivy_gobblin-ivy-download INFO –
{quote}
[Loaded org.apache.log4j.DefaultThrowableRenderer from
file:/export/apps/azkaban/azkaban-exec-server/azkaban-exec-server-1102356.01216_00358/executions/22966290/lib/gobblin-azkaban-0.0.13-all.jar]
[Loaded java.io.LineNumberReader from
/export/apps/jdk/JDK-1_8_0_172/jre/lib/rt.jar]
WARN Serialization class not found:
java.lang.ClassNotFoundException: Class
*org.apache.gobblin.util.WritableShimSerialization* not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2124)
at
org.apache.hadoop.io.serializer.SerializationFactory.add(SerializationFactory.java:71)
at
org.apache.hadoop.io.serializer.SerializationFactory.<init>(SerializationFactory.java:63)
at
org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1238)
at
org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1156)
at
org.apache.hadoop.io.SequenceFile$BlockCompressWriter.<init>(SequenceFile.java:1511)
at
org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:277)
at
org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:372)
at
org.apache.gobblin.metastore.FsStateStore.put(FsStateStore.java:167)
at
org.apache.gobblin.runtime.FsDatasetStateStore.persistDatasetState(FsDatasetStateStore.java:390)
at
org.apache.gobblin.runtime.FsDatasetStateStore.persistDatasetState(FsDatasetStateStore.java:90)
at
org.apache.gobblin.runtime.SafeDatasetCommit.persistDatasetState(SafeDatasetCommit.java:422)
at
org.apache.gobblin.runtime.SafeDatasetCommit.call(SafeDatasetCommit.java:192)
at
org.apache.gobblin.runtime.SafeDatasetCommit.call(SafeDatasetCommit.java:65)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.gobblin.util.executors.MDCPropagatingRunnable.run(MDCPropagatingRunnable.java:39)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[Loaded
org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer from
file:/export/apps/hadoop/hadoop-bin_274273/share/hadoop/common/hadoop-common-2.7.4.273.jar]
[Loaded org.apache.hadoop.io.serializer.avro.AvroReflectSerializable from
file:/export/apps/hadoop/hadoop-bin_274273/share/hadoop/common/hadoop-common-2.7.4.273.jar]
[Loaded com.google.common.base.Throwables from
file:/export/apps/azkaban/azkaban-exec-server/azkaban-exec-server-1102356.01216_00358/executions/22966290/lib/gobblin-azkaban-0.0.13-all.jar]
ERROR Failed to persist dataset state for dataset
[watermark.datetime.1546329600000, watermark.unit.{}] of job
job_zendeskTest_zendeskTicketMetricsChrisTest7_PullTicketMetricsFromZendesk_zendesk_holdem_httpserver-pull_-2010674809_1581127818658
java.io.IOException: Could not find a serializer for the Value class:
'org.apache.gobblin.runtime.JobState.DatasetState'. Please ensure that the
configuration 'io.serializations' is properly configured, if you're usingcustom
serialization.
at
org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1254)
at
org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1156)
at
org.apache.hadoop.io.SequenceFile$BlockCompressWriter.<init>(SequenceFile.java:1511)
at
org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:277)
at
org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:372)
at
org.apache.gobblin.metastore.FsStateStore.put(FsStateStore.java:167)
at
org.apache.gobblin.runtime.FsDatasetStateStore.persistDatasetState(FsDatasetStateStore.java:390)
at
org.apache.gobblin.runtime.FsDatasetStateStore.persistDatasetState(FsDatasetStateStore.java:90)
at
org.apache.gobblin.runtime.SafeDatasetCommit.persistDatasetState(SafeDatasetCommit.java:422)
at
org.apache.gobblin.runtime.SafeDatasetCommit.call(SafeDatasetCommit.java:192)
at
org.apache.gobblin.runtime.SafeDatasetCommit.call(SafeDatasetCommit.java:65)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.gobblin.util.executors.MDCPropagatingRunnable.run(MDCPropagatingRunnable.java:39)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
INFO Attempting to shutdown ExecutorService:
com.google.common.util.concurrent.MoreExecutors$ListeningDecorator@5b9c78
{quote}
was (Author: chris9692):
{quote}[Loaded org.apache.log4j.DefaultThrowableRenderer from
file:/export/apps/azkaban/azkaban-exec-server/azkaban-exec-server-1102356.01216_00358/executions/22966290/lib/gobblin-azkaban-0.0.13-all.jar]
[Loaded java.io.LineNumberReader from
/export/apps/jdk/JDK-1_8_0_172/jre/lib/rt.jar]
WARN Serialization class not found:
java.lang.ClassNotFoundException: Class
*org.apache.gobblin.util.WritableShimSerialization* not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2124)
at
org.apache.hadoop.io.serializer.SerializationFactory.add(SerializationFactory.java:71)
at
org.apache.hadoop.io.serializer.SerializationFactory.<init>(SerializationFactory.java:63)
at
org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1238)
at
org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1156)
at
org.apache.hadoop.io.SequenceFile$BlockCompressWriter.<init>(SequenceFile.java:1511)
at
org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:277)
at
org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:372)
at
org.apache.gobblin.metastore.FsStateStore.put(FsStateStore.java:167)
at
org.apache.gobblin.runtime.FsDatasetStateStore.persistDatasetState(FsDatasetStateStore.java:390)
at
org.apache.gobblin.runtime.FsDatasetStateStore.persistDatasetState(FsDatasetStateStore.java:90)
at
org.apache.gobblin.runtime.SafeDatasetCommit.persistDatasetState(SafeDatasetCommit.java:422)
at
org.apache.gobblin.runtime.SafeDatasetCommit.call(SafeDatasetCommit.java:192)
at
org.apache.gobblin.runtime.SafeDatasetCommit.call(SafeDatasetCommit.java:65)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.gobblin.util.executors.MDCPropagatingRunnable.run(MDCPropagatingRunnable.java:39)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[Loaded
org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer from
file:/export/apps/hadoop/hadoop-bin_274273/share/hadoop/common/hadoop-common-2.7.4.273.jar]
[Loaded org.apache.hadoop.io.serializer.avro.AvroReflectSerializable from
file:/export/apps/hadoop/hadoop-bin_274273/share/hadoop/common/hadoop-common-2.7.4.273.jar]
[Loaded com.google.common.base.Throwables from
file:/export/apps/azkaban/azkaban-exec-server/azkaban-exec-server-1102356.01216_00358/executions/22966290/lib/gobblin-azkaban-0.0.13-all.jar]
ERROR Failed to persist dataset state for dataset
[watermark.datetime.1546329600000, watermark.unit.{}] of job
job_zendeskTest_zendeskTicketMetricsChrisTest7_PullTicketMetricsFromZendesk_zendesk_holdem_httpserver-pull_-2010674809_1581127818658
java.io.IOException: Could not find a serializer for the Value class:
'org.apache.gobblin.runtime.JobState.DatasetState'. Please ensure that the
configuration 'io.serializations' is properly configured, if you're usingcustom
serialization.
at
org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1254)
at
org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1156)
at
org.apache.hadoop.io.SequenceFile$BlockCompressWriter.<init>(SequenceFile.java:1511)
at
org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:277)
at
org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:372)
at
org.apache.gobblin.metastore.FsStateStore.put(FsStateStore.java:167)
at
org.apache.gobblin.runtime.FsDatasetStateStore.persistDatasetState(FsDatasetStateStore.java:390)
at
org.apache.gobblin.runtime.FsDatasetStateStore.persistDatasetState(FsDatasetStateStore.java:90)
at
org.apache.gobblin.runtime.SafeDatasetCommit.persistDatasetState(SafeDatasetCommit.java:422)
at
org.apache.gobblin.runtime.SafeDatasetCommit.call(SafeDatasetCommit.java:192)
at
org.apache.gobblin.runtime.SafeDatasetCommit.call(SafeDatasetCommit.java:65)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.gobblin.util.executors.MDCPropagatingRunnable.run(MDCPropagatingRunnable.java:39)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
INFO Attempting to shutdown ExecutorService:
com.google.common.util.concurrent.MoreExecutors$ListeningDecorator@5b9c78
{quote}
> Gaas jobs not able to write to state store
> ------------------------------------------
>
> Key: GOBBLIN-1055
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1055
> Project: Apache Gobblin
> Issue Type: Bug
> Components: state-management
> Affects Versions: 0.14.0
> Reporter: Chris Li
> Assignee: Hung Tran
> Priority: Major
>
> When state.store.enabled=true and state.store target is on HDFS, Gaas flows
> run into error when writing job state. The reason is that Gaas used a
> particularly GobblinClassLoader to load classes, and without giving that
> information to Hadoop Configuration, the Hadoop hook tries to load the custom
> serializer using default Class Loader.
> See stack trace in comments. While *WritableShimSerialization* was loaded **
> successfully by GobblinClassLoader,
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
