AnthonyCrxHexa commented on issue #8858:
URL: https://github.com/apache/druid/issues/8858#issuecomment-1429518254
> When the exception is thrown, only the following files are being created
in the specified output directory:
>
> ```
> bash-4.2# ls
> druid_dataSource.csv druid_dataSource_raw.csv druid_segments.csv
druid_segments_raw.csv
> ```
>
> Probably interesting, while `druid_segments.csv` contains the correctly
rewritten loadspec, I can see `druid_segments_raw.csv` has the old locations
specified if I decode the hexblob using Pythons `binascii.unhexlify()`.
> I'm trying to migrate my segments from local storage to S3. When running
`export-metadata` with parameters `-b`/`-k` (rewriting deep storage location to
S3), it crashes while exporting the segments table. Exporting without rewriting
works fine.
>
> I suspect it might be due to the fact that not all segments in the table
refer to the same storage location. Recently created and new segments are
actually stored in the target bucket already (storage location in
`common/runtime.conf` was changed to reflect the S3 configuration).
>
> ### Affected Version
> 0.16
>
> ### Description
> ```
> bash-4.2# java -classpath "lib/*"
-Dlog4j.configurationFile=conf/druid/cluster/_common/log4j2.xml
-Ddruid.extensions.directory="extensions" -Ddruid.extensions.loadList=[]
org.apache.druid.cli.Main tools export-metadata --connectURI
"jdbc:derby://localhost:1527/var/druid/metadata.db;" -o /tmp -b druid-bucket -k
druid/segments
> Nov 12, 2019 10:11:27 AM org.hibernate.validator.internal.util.Version
<clinit>
> INFO: HV000001: Hibernate Validator 5.1.3.Final
> 2019-11-12T10:11:27,504 INFO [main]
org.apache.druid.guice.JsonConfigurator - Loaded class[class
org.apache.druid.common.config.NullValueHandlingConfig] from
props[druid.generic.] as
[org.apache.druid.common.config.NullValueHandlingConfig@3da05287]
> 2019-11-12T10:11:27,552 INFO [main]
org.apache.druid.guice.JsonConfigurator - Loaded class[class
org.apache.druid.guice.ExtensionsConfig] from props[druid.extensions.] as
[ExtensionsConfig{searchCurrentClassloader=true, directory='extensions',
useExtensionClassloaderFirst=false,
hadoopDependenciesDir='hadoop-dependencies',
hadoopContainerDruidClasspath='null', addExtensionsToHadoopContainer=false,
loadList=[]}]
> 2019-11-12T10:11:27,640 INFO [main]
org.apache.druid.guice.JsonConfigurator - Loaded class[class
org.apache.druid.guice.ModulesConfig] from props[druid.modules.] as
[ModulesConfig{excludeList=[]}]
> 2019-11-12T10:11:27,795 INFO [main]
org.apache.druid.initialization.Initialization - Adding implementation
[org.apache.druid.common.aws.AWSModule] for class [interface
org.apache.druid.initialization.DruidModule] from classpath extension
> 2019-11-12T10:11:27,797 INFO [main]
org.apache.druid.initialization.Initialization - Adding implementation
[org.apache.druid.common.gcp.GcpModule] for class [interface
org.apache.druid.initialization.DruidModule] from classpath extension
> 2019-11-12T10:11:28,042 INFO [main] org.apache.druid.guice.Jerseys -
Adding Jersey resource: org.apache.druid.server.StatusResource
> 2019-11-12T10:11:28,476 INFO [main]
org.apache.druid.guice.JsonConfigurator - Loaded class[interface
org.apache.druid.server.security.Escalator] from props[druid.escalator.] as
[NoopEscalator{}]
> 2019-11-12T10:11:28,486 INFO [main]
org.apache.druid.guice.JsonConfigurator - Loaded class[class
org.apache.druid.curator.CuratorConfig] from props[druid.zk.service.] as
[org.apache.druid.curator.CuratorConfig@3015db78]
> 2019-11-12T10:11:28,490 INFO [main]
org.apache.druid.guice.JsonConfigurator - Loaded class[class
org.apache.druid.curator.ExhibitorConfig] from props[druid.exhibitor.service.]
as [org.apache.druid.curator.ExhibitorConfig@6b760460]
> 2019-11-12T10:11:28,543 INFO [main] org.apache.curator.utils.Compatibility
- Running in ZooKeeper 3.4.x compatibility mode
> 2019-11-12T10:11:28,545 WARN [main]
org.apache.curator.retry.ExponentialBackoffRetry - maxRetries too large (30).
Pinning to 29
> 2019-11-12T10:11:28,583 INFO [main]
org.apache.druid.guice.JsonConfigurator - Loaded class[class
org.apache.druid.server.initialization.ZkPathsConfig] from
props[druid.zk.paths.] as
[org.apache.druid.server.initialization.ZkPathsConfig@58d3f4be]
> 2019-11-12T10:11:28,591 INFO [main]
org.apache.druid.guice.JsonConfigurator - Loaded class[class
org.apache.druid.server.security.AuthConfig] from props[druid.auth.] as
[AuthConfig{authenticatorChain=null, authorizers=null, unsecuredPaths=[],
allowUnauthenticatedHttpOptions=false}]
> 2019-11-12T10:11:28,642 INFO [main]
org.apache.druid.guice.JsonConfigurator - Loaded class[class
org.apache.druid.server.metrics.DruidMonitorSchedulerConfig] from
props[druid.monitoring.] as
[org.apache.druid.server.metrics.DruidMonitorSchedulerConfig@7bc6d27a]
> 2019-11-12T10:11:28,646 INFO [main]
org.apache.druid.guice.JsonConfigurator - Loaded class[class
org.apache.druid.server.metrics.MonitorsConfig] from props[druid.monitoring.]
as [MonitorsConfig{monitors=[]}]
> 2019-11-12T10:11:28,647 INFO [main]
org.apache.druid.server.emitter.EmitterModule - Underlying emitter for
ServiceEmitter: org.apache.druid.java.util.emitter.core.NoopEmitter@1ac45389
> 2019-11-12T10:11:28,647 INFO [main]
org.apache.druid.server.emitter.EmitterModule - Extra service dimensions:
{version=0.16.0-incubating}
> 2019-11-12T10:11:28,648 INFO [main]
org.apache.druid.server.metrics.MetricsModule - Adding
monitor[org.apache.druid.query.ExecutorServiceMonitor@10bd9df0]
> 2019-11-12T10:11:28,648 INFO [main]
org.apache.druid.server.metrics.MetricsModule - Adding
monitor[org.apache.druid.server.initialization.jetty.JettyServerModule$JettyMonitor@732c9b5c]
> 2019-11-12T10:11:28,720 INFO [main]
org.apache.druid.metadata.storage.derby.DerbyConnector - Derby connector
instantiated with metadata storage
[org.apache.druid.metadata.NoopMetadataStorageProvider$1].
> 2019-11-12T10:11:28,720 INFO [main] org.apache.druid.cli.ExportMetadata -
Exporting datasource table: druid_dataSource
> 2019-11-12T10:11:28,862 INFO [main] org.apache.druid.cli.ExportMetadata -
Exporting segments table: druid_segments
> Exception in thread "main" java.lang.NullPointerException
> at
org.apache.druid.cli.ExportMetadata.getUniqueIDFromLocalLoadSpec(ExportMetadata.java:537)
> at
org.apache.druid.cli.ExportMetadata.makePayloadWithConvertedLoadSpec(ExportMetadata.java:429)
> at
org.apache.druid.cli.ExportMetadata.rewriteSegmentsExport(ExportMetadata.java:407)
> at org.apache.druid.cli.ExportMetadata.run(ExportMetadata.java:225)
> at org.apache.druid.cli.Main.main(Main.java:113)
> ```
HI, exact same problem here on latest version, did you figured it out ?
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
