i added java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper
from previous email and added to ignite lib. after adding this library
ignite node started but i have encounter another error just like below:
[13:03:28] __________ ________________
[13:03:28] / _/ ___/ |/ / _/_ __/ __/
[13:03:28] _/ // (7 7 // / / / / _/
[13:03:28] /___/\___/_/|_/___/ /_/ /___/
[13:03:28]
[13:03:28] ver. 2.6.0#20180710-sha1:669feacc
[13:03:28] 2018 Copyright(C) Apache Software Foundation
[13:03:28]
[13:03:28] Ignite documentation: http://ignite.apache.org
[13:03:28]
[13:03:28] Quiet mode.
[13:03:28] ^-- Logging to file
'/usr/local/apache-ignite-fabric-2.6.0-bin/work/log/ignite-41a0490a.log'
[13:03:28] ^-- Logging by 'Log4JLogger [quiet=true,
config=/usr/local/apache-ignite-fabric-2.6.0-bin/config/ignite-log4j.xml]'
[13:03:28] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false
or "-v" to ignite.{sh|bat}
[13:03:28]
[13:03:28] OS: Linux 4.15.0-46-generic amd64
[13:03:28] VM information: Java(TM) SE Runtime Environment 1.8.0_192-ea-b04
Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.192-b04
[13:03:28] Configured plugins:
[13:03:28] ^-- Ignite Native I/O Plugin [Direct I/O]
[13:03:28] ^-- Copyright(C) Apache Software Foundation
[13:03:28]
[13:03:28] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler
[tryStop=false, timeout=0]]
[13:03:28] Message queue limit is set to 0 which may lead to potential
OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due
to message queues growth on sender and receiver sides.
[13:03:29] Security status [authentication=off, tls/ssl=off]
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/local/apache-ignite-fabric-2.6.0-bin/libs/slf4j-nop-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/apache-ignite-fabric-2.6.0-bin/libs/htrace%20dependency/slf4j-nop-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/apache-ignite-fabric-2.6.0-bin/libs/ignite-rest-http/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/apache-ignite-fabric-2.6.0-bin/libs/ignite-yarn/ignite-yarn-2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/apache-ignite-fabric-2.6.0-bin/libs/ignite-zookeeper/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.helpers.NOPLoggerFactory]
[13:03:31] HADOOP_HOME is set to /usr/local/hadoop
[13:03:31] Resolved Hadoop classpath locations:
/usr/local/hadoop/share/hadoop/common, /usr/local/hadoop/share/hadoop/hdfs,
/usr/local/hadoop/share/hadoop/mapreduce
[13:03:32] Nodes started on local machine require more than 20% of physical
RAM what can lead to significant slowdown due to swapping (please decrease
JVM heap size, data region size or checkpoint buffer size)
[required=5344MB, available=7953MB]
[13:03:34] Performance suggestions for grid (fix if possible)
[13:03:34] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[13:03:34] ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM
options)
[13:03:34] ^-- Set max direct memory size if getting 'OOME: Direct buffer
memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options)
[13:03:34] ^-- Disable processing of calls to System.gc() (add
'-XX:+DisableExplicitGC' to JVM options)
[13:03:34] Refer to this page for more performance suggestions:
https://apacheignite.readme.io/docs/jvm-and-system-tuning
[13:03:34]
[13:03:34] To start Console Management & Monitoring run
ignitevisorcmd.{sh|bat}
[13:03:34]
[13:03:34] Ignite node started OK (id=41a0490a)
[13:03:34] Topology snapshot [ver=4, servers=2, clients=0, CPUs=8,
offheap=3.1GB, heap=2.0GB]
[13:03:34] ^-- Node [id=41A0490A-8A7B-4977-A48B-B5D98D49CF1B,
clusterState=ACTIVE]
[13:03:34] Data Regions Configured:
[13:03:34] ^-- default [initSize=256.0 MiB, maxSize=1.6 GiB,
persistenceEnabled=false]
[13:03:42] New version is available at ignite.apache.org: 2.7.0
[13:03:56] Topology snapshot [ver=5, servers=1, clients=0, CPUs=8,
offheap=1.6GB, heap=1.0GB]
[13:03:56] ^-- Node [id=41A0490A-8A7B-4977-A48B-B5D98D49CF1B,
clusterState=ACTIVE]
[13:03:56] Data Regions Configured:
[13:03:56] ^-- default [initSize=256.0 MiB, maxSize=1.6 GiB,
persistenceEnabled=false]
[2019-03-12 13:04:08,878][ERROR][igfs-igfs-ipc-#64][IgfsImpl] File info
operation in DUAL mode failed [path=/output]
class org.apache.ignite.IgniteException: For input string: "30s"
at
org.apache.ignite.internal.processors.hadoop.impl.fs.HadoopLazyConcurrentMap.getOrCreate(HadoopLazyConcurrentMap.java:100)
at
org.apache.ignite.internal.processors.hadoop.impl.delegate.HadoopCachingFileSystemFactoryDelegate.getWithMappedName(HadoopCachingFileSystemFactoryDelegate.java:53)
at
org.apache.ignite.internal.processors.hadoop.impl.delegate.HadoopBasicFileSystemFactoryDelegate.get(HadoopBasicFileSystemFactoryDelegate.java:75)
at
org.apache.ignite.internal.processors.hadoop.impl.delegate.HadoopBasicFileSystemFactoryDelegate.get(HadoopBasicFileSystemFactoryDelegate.java:43)
at
org.apache.ignite.internal.processors.hadoop.impl.delegate.HadoopIgfsSecondaryFileSystemDelegateImpl.fileSystemForUser(HadoopIgfsSecondaryFileSystemDelegateImpl.java:517)
at
org.apache.ignite.internal.processors.hadoop.impl.delegate.HadoopIgfsSecondaryFileSystemDelegateImpl.info
(HadoopIgfsSecondaryFileSystemDelegateImpl.java:296)
at org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem.info
(IgniteHadoopIgfsSecondaryFileSystem.java:240)
at
org.apache.ignite.internal.processors.igfs.IgfsImpl.resolveFileInfo(IgfsImpl.java:1600)
at
org.apache.ignite.internal.processors.igfs.IgfsImpl.access$800(IgfsImpl.java:110)
at
org.apache.ignite.internal.processors.igfs.IgfsImpl$6.call(IgfsImpl.java:524)
at
org.apache.ignite.internal.processors.igfs.IgfsImpl$6.call(IgfsImpl.java:517)
at
org.apache.ignite.internal.processors.igfs.IgfsImpl.safeOp(IgfsImpl.java:1756)
at org.apache.ignite.internal.processors.igfs.IgfsImpl.info
(IgfsImpl.java:517)
at
org.apache.ignite.internal.processors.igfs.IgfsIpcHandler$2.apply(IgfsIpcHandler.java:341)
at
org.apache.ignite.internal.processors.igfs.IgfsIpcHandler$2.apply(IgfsIpcHandler.java:332)
at org.apache.ignite.igfs.IgfsUserContext.doAs(IgfsUserContext.java:54)
at
org.apache.ignite.internal.processors.igfs.IgfsIpcHandler.processPathControlRequest(IgfsIpcHandler.java:332)
at
org.apache.ignite.internal.processors.igfs.IgfsIpcHandler.execute(IgfsIpcHandler.java:241)
at
org.apache.ignite.internal.processors.igfs.IgfsIpcHandler.access$000(IgfsIpcHandler.java:57)
at
org.apache.ignite.internal.processors.igfs.IgfsIpcHandler$1.run(IgfsIpcHandler.java:167)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.IgniteCheckedException: For input
string: "30s"
at
org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7307)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:259)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:171)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
at
org.apache.ignite.internal.processors.hadoop.impl.fs.HadoopLazyConcurrentMap$ValueWrapper.getValue(HadoopLazyConcurrentMap.java:191)
at
org.apache.ignite.internal.processors.hadoop.impl.fs.HadoopLazyConcurrentMap.getOrCreate(HadoopLazyConcurrentMap.java:93)
... 22 more
Caused by: java.lang.NumberFormatException: For input string: "30s"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:589)
at java.lang.Long.parseLong(Long.java:631)
at org.apache.hadoop.conf.Configuration.getLong(Configuration.java:1538)
at org.apache.hadoop.hdfs.DFSClient$Conf.<init>(DFSClient.java:430)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:540)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:524)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:146)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:476)
at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:217)
at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:214)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:214)
at
org.apache.ignite.internal.processors.hadoop.impl.delegate.HadoopBasicFileSystemFactoryDelegate.create(HadoopBasicFileSystemFactoryDelegate.java:117)
at
org.apache.ignite.internal.processors.hadoop.impl.delegate.HadoopBasicFileSystemFactoryDelegate.getWithMappedName(HadoopBasicFileSystemFactoryDelegate.java:95)
at
org.apache.ignite.internal.processors.hadoop.impl.delegate.HadoopCachingFileSystemFactoryDelegate.access$001(HadoopCachingFileSystemFactoryDelegate.java:32)
at
org.apache.ignite.internal.processors.hadoop.impl.delegate.HadoopCachingFileSystemFactoryDelegate$1.createValue(HadoopCachingFileSystemFactoryDelegate.java:37)
at
org.apache.ignite.internal.processors.hadoop.impl.delegate.HadoopCachingFileSystemFactoryDelegate$1.createValue(HadoopCachingFileSystemFactoryDelegate.java:35)
at
org.apache.ignite.internal.processors.hadoop.impl.fs.HadoopLazyConcurrentMap$ValueWrapper.init(HadoopLazyConcurrentMap.java:173)
at
org.apache.ignite.internal.processors.hadoop.impl.fs.HadoopLazyConcurrentMap$ValueWrapper.access$100(HadoopLazyConcurrentMap.java:154)
at
org.apache.ignite.internal.processors.hadoop.impl.fs.HadoopLazyConcurrentMap.getOrCreate(HadoopLazyConcurrentMap.java:82)
... 22 more
============================================
i have three exm configuration file just like beow, i have execute hadoop
with below command with config switch
time hadoop --config
/home/mehdi/ignite-conf/ignite-configs-master/igfs-hadoop-fs-cache/ignite_conf
jar
/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar
wordcount /user/input/ /output
i have created /user/input directory in hdfs and i want to put result in
output folder.
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>ignite</value>
</property>
<property>
<name>mapreduce.jobtracker.address</name>
<value>127.0.0.1:11211</value>
</property>
</configuration>
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration><?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/app/hadoop/tmp</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>fs.default.name</name>
<value>igfs://igfs@</value>
</property>
<property>
<!-- FS driver class for the 'igfs://' URIs. -->
<name>fs.igfs.impl</name>
<value>org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem</value>
</property>
<property>
<!-- FS driver class for the 'igfs://' URIs in Hadoop2.x -->
<name>fs.AbstractFileSystem.igfs.impl</name>
<value>org.apache.ignite.hadoop.fs.v2.IgniteHadoopFileSystem</value>
</property>
</configuration>
