I use hbase-0.94 and hadoop-2.2, there is below exception:
2014-07-04 12:43:49,700 WARN org.apache.hadoop.hdfs.DFSClient: failed to
connect to
DomainSocket(fd=322,path=/home/hadoop/hadoop-current/cdh4-dn-socket/dn_socket)
java.net.SocketTimeoutException: read(2) error: Resource temporarily
I use hadoop-2.4, I want use the hdfs cache function.
I use ulimit -l 32212254720 linux command to set size of max locked
memory, but there is below error:
ulimit -l 322
-bash: ulimit: max locked memory: cannot modify limit: Operation not
permitted
How can I set size of max locked memory?
On April 11 hadoop-2.4 is released, the hadoop-2.4 does not include
heterogeneous storages function, when does hadoop include the function?
Thanks,
LiuLei
there?
Ascot
On 14 Apr, 2014, at 4:34 pm, lei liu liulei...@gmail.com wrote:
On April 11 hadoop-2.4 is released, the hadoop-2.4 does not include
heterogeneous storages function, when does hadoop include the function?
Thanks,
LiuLei
Hadoop-2.4 is release, where can I download the hadoop-2.4 code from?
Thanks,
LiuLei
I use Hbase-0.94 and hadoop-2.0.
I install one HDFS cluster that have 15 datanodes. If network bandwidth of
two datanodes is saturation(example 100m/s), writing performance of the entire
hdfs cluster is slow.
I think that the slow datanodes affect the writing performance of the entire
cluster.
I use Hbase0.94 and CDH4. There are 25729 tcp connections in one
machine,example:
hadoop@apayhbs081 ~ $ netstat -a | wc -l
25729
The linux configration is :
softcore0
hardrss 1
hardnproc 20
softnproc 20
I use HBase0.96 and CDH4.3.1.
I use Short-Circuit Local Read:
property
namedfs.client.read.shortcircuit/name
valuetrue/value/propertyproperty
namedfs.domain.socket.path/name
value/home/hadoop/cdh4-dn-socket/dn_socket/value/property
When one disk is bad, because the RegionServer open
When I use the hadoop security, I must use jsvc to start datanode. Why must
use jsvc to start datanode? What are the advantages do that?
Thanks,
LiuLei
There is DelegationToken in hadoop2. What is the role of DelegationToken
and how to use the DelegationToken ?
Thanks,
LiuLei
In CDH3u5, when the DataNode is Decommissioned, the DataNode progress will
be shutdown by NameNode.
But In CDH4.3.1, when the DataNode is Decommissioned, the DataNode progress
will be not shutdown by NameNode.
When the datanode is Decommissioned, why the datanode is not automatically
shutdown
In CDH3u3 there is
ClientDatanodeProtocoleclipse-javadoc:%E2%98%82=hadoop-0.20.2-cdh3u5_core/src%5C/hdfs%3Corg.apache.hadoop.hdfs.protocol%7BClientDatanodeProtocol.java%E2%98%83ClientDatanodeProtocol.recoverBlock
method, the method is used to recover block when data streaming is failed.
But in
I use CDH4.3.1, When I start NameNode,and transition one NameNode to
active, there is below deadlock:
Found one Java-level deadlock:
=
22558696@qtp-1616586953-6:
waiting to lock monitor 0x2aaab3621f40 (object 0xf7646958, a
I use CDH4.3.1 and run the TestHDFSCLI unit test,but there are below errors:
2013-10-10 13:05:39,671 INFO cli.CLITestHelper
(CLITestHelper.java:displayResults(156)) -
---
2013-10-10 13:05:39,671 INFO cli.CLITestHelper
I use CDH-4.3.1, When I start datanode, there are below error:
2013-09-26 17:57:07,803 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server at
0.0.0.0:40075
2013-09-26 17:57:07,814 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: dfs.webhdfs.enabled = false
2013-09-26
I use the CDH-4.3.1 and mr1, when I run one job, I am getting the following
error.
Exception in thread main java.lang.IncompatibleClassChangeError:
Found interface org.apache.hadoop.mapreduce.JobContext, but class was
expected
at
was
compiled against vs the version that you're running against. Make sure that
you're not packaging hadoop jar's into your jar and make sure you're
compiling against the correct version as well.
On Sun, Sep 29, 2013 at 7:27 PM, lei liu liulei...@gmail.com wrote:
I use the CDH-4.3.1 and mr1
I use the CDH-4.3.1 and mr1, when I run one job, I am getting the following
error.
Exception in thread main java.lang.IncompatibleClassChangeError:
Found interface org.apache.hadoop.mapreduce.JobContext, but class was
expected
at
Hello，
Can anybody answer the question?
2013/9/1 lei liu liulei...@gmail.com
Hi Jitendr, thanks for your reply.
If MutableCounterLong is uesed to IO/sec statistics, I think the value of
MutableCounterLong
should be divided by 10 and be reseted to zero per ten seconds
that. I missed anything please tell me. Looking forward to your reply.
Thanks,
LiuLei
2013/9/1 Jitendra Yadav jeetuyadav200...@gmail.com
Yes, MutableCounterLong helps to gather DataNode read/write statics.
There is more option available within this metric
Regards
Jitendra
On 8/31/13, lei liu
thread
number I'm not bit sure right now.
Thanks
Jiitendra
On Fri, Aug 30, 2013 at 1:40 PM, lei liu liulei...@gmail.com wrote:
Hi Jitendra,
If I want to statistics number of bytes read per second,and display the
result into ganglia, should I use MutableCounterLong or MutableGaugeLong
I use QJM, do I need to config two directories for the
dfs.namenode.name.dir, one local filesystem path and one NFS path?
I think the Stadnby NameNode also store the fsimage, so I think I only
need to config one local file system path.
Thanks,
LiuLei
I use the metrics v2, there are COUNTER and GAUGE metric type in metrics
v2.
What is the difference between the two？
Thanks,
LiuLei
,
LiuLei
2013/8/30 Jitendra Yadav jeetuyadav200...@gmail.com
Hi,
Below link contains the answer for your question.
http://hadoop.apache.org/docs/r1.2.0/api/org/apache/hadoop/metrics2/package-summary.html
Regards
Jitendra
On Fri, Aug 30, 2013 at 11:35 AM, lei liu liulei...@gmail.com
There are dfs.client.read.shortcircuit and
dfs.client.domain.socket.data.traffic configuration in domain socket. What
is different them?
Thanks,
LiuLei
I use hadoop2 and hbase0.94, but there is below exception:
2013-08-28 11:36:12,922 ERROR
[MASTER_TABLE_OPERATIONS-dw74.kgb.sqa.cm4,13646,1377660964832-0]
executor.EventHandler(172): Caught throwable while processing
event C_M_DELETE_TABLE
java.lang.IllegalArgumentException: Wrong FS: file:/tmp/
, lei liu liulei...@gmail.com wrote:
I use hadoop2 and hbase0.94, but there is below exception:
2013-08-28 11:36:12,922 ERROR
[MASTER_TABLE_OPERATIONS-dw74.kgb.sqa.cm4,13646,1377660964832-0]
executor.EventHandler(172): Caught throwable while processing
event C_M_DELETE_TABLE
In org.apache.hadoop.hbase.coprocessor.TestMasterObserver unit test.
2013/8/28 lei liu liulei...@gmail.com
When I run hbase unit test, there is the exception.
2013/8/28 Harsh J ha...@cloudera.com
Moving to u...@hbase.apache.org.
Please share your hbase-site.xml and core-site.xml
, 2013 at 11:39 AM, lei liu liulei...@gmail.com wrote:
I set the dfs.image.transfer.bandwidthPerSec. to 50M, and the
performance is below:
2013-08-14 12:32:33,079 INFO my.EditLogPerformance: totalCount:1342440
speed:
2013-08-14 12:32:43,082 INFO my.EditLogPerformance: totalCount:1363338
There is ReconfigurationServlet class in hadoop-2.0.5.
How I to use the function for NameNode and DataNode?
Thanks,
LiuLei
The fsimage file size is 1658934155
2013/8/13 Harsh J ha...@cloudera.com
How large are your checkpointed fsimage files?
On Mon, Aug 12, 2013 at 3:42 PM, lei liu liulei...@gmail.com wrote:
When Standby Namenode is doing checkpoint, upload the image file to
Active
NameNode, the Active
at 11:52 AM, lei liu liulei...@gmail.com wrote:
The fsimage file size is 1658934155
2013/8/13 Harsh J ha...@cloudera.com
How large are your checkpointed fsimage files?
On Mon, Aug 12, 2013 at 3:42 PM, lei liu liulei...@gmail.com wrote:
When Standby Namenode is doing checkpoint, upload
When Standby Namenode is doing checkpoint, upload the image file to Active
NameNode, the Active NameNode is very slow. What is reason result to the
Active NameNode is slow?
Thanks,
LiuLei
.
On Wed, Aug 7, 2013 at 4:27 PM, lei liu liulei...@gmail.com wrote:
I use hadoop-2.0.5 and config hadoop-metrics2.properties file with below
content.
*.sink.ganglia.class=org.
apache.hadoop.metrics2.sink.ganglia.GangliaSink31
*.sink.ganglia.period=10
*.sink.ganglia.supportsparse=true
I use hadoop-2.0.5, there are MutableCounterLong and MutableCounterLong
class in metrics v2.
I am studing metrics v2 code.
What are difference MutableCounterLong and MutableCounterLong class ?
I find the MutableCounterLong is used to calculate throughput, is that
right? How does the metrics
I use hadoop-2.0.5 and config hadoop-metrics2.properties file with below
content.
*.sink.ganglia.class=org.
apache.hadoop.metrics2.sink.ganglia.GangliaSink31
*.sink.ganglia.period=10
*.sink.ganglia.supportsparse=true
namenode.sink.ganglia.servers=10.232.98.74:8649
I use hadoop-2.0.5 and config hadoop-metrics2.properties file with below
content.
*.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31
*.sink.ganglia.period=10
*.sink.ganglia.supportsparse=true
namenode.sink.ganglia.servers=10.232.98.74:8649
There is code in MutableRate class:
public synchronized void snapshot(MetricsRecordBuilder builder, boolean
all) {
if (all || changed()) {
numSamples += intervalStat.numSamples();
builder.addCounter(numInfo, numSamples)
.addGauge(avgInfo, lastStat().mean());
does ganglia to display the value of MutableCounterLong class? Is
that fsyncCount.value or fsyncCount.value/10?
2013/8/6 lei liu liulei...@gmail.com
I use hadoop-2.0.5 and config hadoop-metrics2.properties file with below
content.
*.sink.ganglia.class=org.apache.hadoop.metrics2
Is the the value of MutableCounterLong class set to zreo per 10 seconds?
2013/8/6 lei liu liulei...@gmail.com
There is @Metric MutableCounterLong fsyncCount metrics in
DataNodeMetrics, the MutableCounterLong class continuously increase the
value, so I think the value in ganglia should
Is the the value of MutableCounterLong class set to zero per 10 seconds?
2013/8/6 lei liu liulei...@gmail.com
Is the the value of MutableCounterLong class set to zreo per 10 seconds?
2013/8/6 lei liu liulei...@gmail.com
There is @Metric MutableCounterLong fsyncCount metrics
lei liu liulei...@gmail.com
Can I use metrics v1 in hadoop-2.0.5?
Thanks,
LiuLei
Can I use metrics v1 in hadoop-2.0.5?
Thanks,
LiuLei
I use hadoop-2.0.5, and QJM for HA.
When Standby NameNode do checkpoint,there are below exception in Standby
NameNode:
2013-08-01 13:43:07,965 INFO
org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer: Triggering
checkpoint because there have been 763426 txns since the last checkpoint,
:17 PM, lei liu liulei...@gmail.com wrote:
I use hadoop-2.0.5 version and use QJM for HA.
I use ./hdfs namenode -bootstrapStandby for StandbyNameNode, but report
below error:
=
About to bootstrap Standby ID nn2 from:
Nameservice
I use hadoop-2.0.5 version and use QJM for HA.
I use ./hdfs namenode -bootstrapStandby for StandbyNameNode, but report
below error:
=
About to bootstrap Standby ID nn2 from:
Nameservice ID: mycluster
Other Namenode ID: nn1
When I use QJM for HA, do I need to save edit log on the local filesystem?
I think the QJM is high availability for edit log, so I don't need to
configuration the dfs.namenode.edits.dir.
Thanks,
LiuLei
I have two namespaces, example below:
property
namedfs.nameservices/name
valuens1,ns2/value
/property
Can I config the dfs.namenode.shared.edits.dir to below content?
property
namedfs.namenode.shared.edits.dir/name
Thanks Harsh.
2013/7/17 Harsh J ha...@cloudera.com
This has been asked previously. Use suffixes to solve your issue. See
http://search-hadoop.com/m/Fingkg6Dk91
On Wed, Jul 17, 2013 at 1:33 PM, lei liu liulei...@gmail.com wrote:
I have two namespaces, example below:
property
I have another question for QJM.
If I use QJM for HA, do I need to save edit log on the local filesystem?
I think the QJM is high availability for edit log, so I don't need to
config the dfs.namenode.edits.dir.
Thanks,
LiuLei
2013/7/17 lei liu liulei...@gmail.com
Thanks Harsh.
2013
I use hadoop-2.0.3. I find when on block is over-replicated, the replicas
to be add to excessReplicateMap attribute of Blockmanager. But when the
block is deleted or the block has the intended number of replicas, the
replicas is not deleted form excessReplicateMap attribute.
I think this is bug.
to transfer
data to DataNodes.
And I guess you can understand Chinese, so I recommend you to read one of
my blog(http://yanbohappy.sinaapp.com/?p=143) and it explain the write
workflow detail.
2013/3/29 lei liu liulei...@gmail.com
Thanks Yanbo for your reply.
I test code
When client write data, if there are three replicates, the sync method
latency time formula should be:
sync method latency time = first datanode receive data time + sencond
datanode receive data time + third datanode receive data time.
if the three datanode receive data time all are 2
.
It's parallel computing rather than serial computing in a sense.
2013/3/28 lei liu liulei...@gmail.com
When client write data, if there are three replicates, the sync method
latency time formula should be:
sync method latency time = first datanode receive data time + sencond
datanode receive
I am using hadoop0.20.2, now I want to use HDFS HA function. I research
AvatarNode. I find if the StandbyNN do checkpoint fail, when next time the
StandbyNN do checkpoint, the same edits file is loaded again. Can same
edits file be loaded more than once in hadoop0.20.2?
if not, what is the harm?
generally be idempotent. In
particular applications, you might find it is OK to treat them as such, but
there are definitely applications where they are not idempotent.
On Sun, Oct 28, 2012 at 2:40 AM, lei liu liulei...@gmail.com wrote:
I think these methods should are idempotent
Hi Steve，
Thank you for your detailed and patiently answered. I understand that.
2012/11/5 Steve Loughran ste...@hortonworks.com
On 4 November 2012 17:25, lei liu liulei...@gmail.com wrote:
I want to know what applications are idempotent or not idempotent? and
Why? Could you give me
I think these methods should are idempotent, these methods should be repeated
calls to be harmless by same client.
Thanks,
LiuLei
, mkdirs, rename and delete cannot generally be idempotent. In
particular applications, you might find it is OK to treat them as such, but
there are definitely applications where they are not idempotent.
On Sun, Oct 28, 2012 at 2:40 AM, lei liu liulei...@gmail.com wrote:
I think
I use NFS V4 to test the java FileLock.
The 192.168.1.233 machine is NFS Server, the nfs configuration are
/home/hdfs.ha/share 192.168.1.221(rw,sync,no_root_squash)
/home/hdfs.ha/share 192.168.1.222(rw,sync,no_root_squash)
in /etc/exports file.
I run below commands to start nfs server:
We are using NFS for Shared storage, Can we use linux nfslcok service to
implement IO Fencing ?
2012/10/26 Steve Loughran ste...@hortonworks.com
On 25 October 2012 14:08, Todd Lipcon t...@cloudera.com wrote:
Hi Liu,
Locks are not sufficient, because there is no way to enforce a lock in a
http://blog.csdn.net/onlyqi/article/details/6544989
https://issues.apache.org/jira/browse/HDFS-2185
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailability.html
http://blog.csdn.net/chenpingbupt/article/details/7922042
I use DistributedCache to add two files to class path, exampe below code :
String jeJarPath = /group/aladdin/lib/je-4.1.7.jar;
DistributedCache.addFileToClassPath(new Path(jeJarPath), conf);
String tairJarPath = /group/aladdin/lib/tair-aladdin-2.3.1.jar
I want to use hadoop to create Berkeley DB index, so I need create one
directory to store Berkeley DB index, There are below code in reduce :
String tmp = job.get(hadoop.tmp.dir);
String shardName = shard + this.shardNum + _ +
UUID.randomUUID().toString();
There are two input direcoties:/user/test1/ and /user/test2/ , I want to
join the two direcoties content, in order to join the two directories, I
need to identity the content are handled by mapper from which directory, so
I use below code in mapper:
private int tag = -1;
@Override
I define the counter to count the bad records, there is below code in map
task;
reporter.incrCounter(bad',
records', 1),
When the job is completed, the pritnt the result to use below code:
long total = counters.findCounter(bad,records).getCounter();
But I have two questions about the counter:
I use hive0.6 version and execute 'select INPUT_FILE_NAME,
BLOCK_OFFSET_INSIDE_FILE from person1' statement, hive0.6 throws below
error:
FAILED: Error in semantic analysis: line 1:7 Invalid Table Alias or Column
Reference INPUT_FILE_NAME error.
Don't hive0.6 support virtual columns?
I use hive0.6 ,I want to create index on one table, how can I do ti?
I know the describe statement, the statement don't display the FIELDS
TERMINATED and LINES TERMINATED, it only display column name and column
type.
2010/9/19 Ted Yu yuzhih...@gmail.com
See bottom of http://wiki.apache.org/hadoop/Hive/LanguageManual/DDL
On Sat, Sep 18, 2010 at 7:13 PM, lei
I use below statement to create one tabale and add one partition:
create external table test(userid bigint,name string, age int) partitioned
by(pt string);
alter table test add partition(pt='01');
Now there is one file in HDFS, the file path is /user/hive/warehouse/user, I
use load statement to
I use ./hive --service metastore command to start metastore server, how to
connection metastore server with hive JDBC client?
I use below statement to create one table:
CREATE TABLE page_view(viewTime INT, userid BIGINT,
page_url STRING, referrer_url STRING,
ip STRING COMMENT 'IP Address of the User')
COMMENT 'This is the page view table'
PARTITIONED BY(dt STRING, country STRING)
ROW FORMAT DELIMITED
I use below statement to create one table:
CREATE TABLE page_view(viewTime INT, userid BIGINT,
page_url STRING, referrer_url STRING,
ip STRING COMMENT 'IP Address of the User')
COMMENT 'This is the page view table'
PARTITIONED BY(dt STRING, country STRING)
ROW FORMAT DELIMITED
I find GroupByOperator cache the Aggregation results of different keys.
Please look below cod:
AggregationBuffer[] aggs = null;
boolean newEntryForHashAggr = false;
keyProber.hashcode = newKeys.hashCode();
// use this to probe the hashmap
keyProber.keys = newKeys;
//
I use hive-0.6 an use mysql as metasore, but hive don't connection the
mysql.
2010-08-30 13:28:24,982 ERROR [main] util.Log4JLogger(125): Failed
initialising database.
Invalid URL: jdbc:mysql://127.0.0.1:3306/hive6?createDatabaseIfNotExist=true
When hadoop one job which is submmited by hive, hadoop need the
hive_exec.jar, hwo to hive add hive_exec.jar to hadoop?
Please tell me the where are codes in hive.
Thanks,
LiuLei
--service hiveserver
*
and then run the code
lei liu wrote:
Hello everyone,
I use JDBC to connection the hive server, sometime I receive below
exception:
java.sql.SQLException: org.apache.thrift.transport.TTransportException:
Cannot read. Remote side has closed. Tried to read 1 bytes
Hi shangan,
You need to set linux coding is UTF-8.
2010/8/16 shangan shan...@corp.kaixin001.com
the fact is that even I hava data in UTF-8 using simplified Chinese, then
doing a select * it will return an unreadable result. Does that mean hive
can only support ascii character ?
2010-08-16
this – one
would think it should work if hadoop does – but we would have to verify.
(ie. yes – it should be possible in theory – but in practice – there are
probably bugs that need to get sorted out for this to happen).
--
*From:* lei liu [mailto:liulei
what is difference hive local model and standalone model. Can the hive local
model be ran in windows?
to 'jdbc:hive://' and run the
example script. This will run the code, which will start Hive in embedded
mode, create a table, do some operations on it, and then drop it.
On Tue, Aug 10, 2010 at 8:05 AM, lei liu liulei...@gmail.com wrote:
Can anybody answer the question?
Thanks,
LiuLei
2010/8/10
Thank you for your reply.
Could you tell me why it is slower if the two paremeters are true and how
slow it is?
2010/8/10 Namit Jain nj...@facebook.com
Yes, it will try to run another map-reduce job to merge the files
From: lei liu [liulei
Hello everyone,
Could everybody tell me how to call UDF/UDAF in hive?
Could you tell me whether the query is slower if I two parameters both are
true?
2010/8/9 Namit Jain nj...@facebook.com
That's right
From: lei liu [liulei...@gmail.com]
Sent: Sunday, August 08, 2010 7:18 PM
To: hive-user@hadoop.apache.org
Subject
I look see below content in
http://wiki.apache.org/hadoop/Hive/HiveClientpage: For embedded mode,
uri is just jdbc:hive://. How can I use JDBC
client embedded mode? Could anybody give me an example?
(hive.merge.mapfiles, true),
HIVEMERGEMAPREDFILES(hive.merge.mapredfiles, false),
Set the above parameters to true before your query.
From: lei liu [liulei...@gmail.com]
Sent: Thursday, August 05, 2010 8:47 PM
To: hive-user@hadoop.apache.org
Subject: How to merge
How can I use the embedded mode of JDBC, could anybody give me an example?
how can I debug code in org.apache.hadoop.hive.ql.exec package?
, 2010 at 6:10 AM, lei liu liulei...@gmail.com wrote:
Because my company reuire we use 0.4.1 version, the version don't
support IN
clause. I want to use the OR clause(example:where id=1 or id=2 or
id=3) to
implement the IN clause(example: id in(1,2,3) ). I know it will be
slower
especially
I have used 'Remote Java Application' in eclipse to debug hive code, now I
want to debug hive and hadoop together, how can I do it?
Thanks,
LiuLei
When I run below sql: INSERT OVERWRITE TABLE tablename1
select_statement1 FROM from_statement, there are many files which size
is zero are stored to hadoop,
How can I merge these small files?
Thanks,
LiuLei
Because my company reuire we use 0.4.1 version, the version don't support IN
clause. I want to use the OR clause(example:where id=1 or id=2 or id=3) to
implement the IN clause(example: id in(1,2,3) ). I know it will be slower
especially when the list after in is very long. Could anybody can
Hello Edward Capriolo,
Thank you for your reply. Are you sure that if you string enough 'or'
together (say 8000) the query parser which uses java beans serialization
will OOM? How many memory you assign to hive?
2010/8/4 Edward Capriolo edlinuxg...@gmail.com
On Wed, Aug 4, 2010 at 6:10 AM, lei
Now I assign 100M memory to hive, you consider that can support how many
'OR' string?
2010/8/5 Edward Capriolo edlinuxg...@gmail.com
On Wed, Aug 4, 2010 at 12:15 PM, lei liu liulei...@gmail.com wrote:
Hello Edward Capriolo,
Thank you for your reply. Are you sure that if you string enough
94 matches
Mail list logo
